DealTracker - Architecture
DealTracker's Architecture
DealTracker's architecture is designed in several parts, many of which are hosted at home in my homelab.
1- The Data Access Layer
The goal of this part is to collect data from second-hand listings published on different marketplaces. This data is restructured and normalized, to allow equivalent processing regardless of the marketplace on which the listing was originally published.
2- The AI Lab
I think building this AI lab is one of my most significant achievements on DealTracker. Its technological maturity comes from a lot of trial and error, restructuring both the training process and the code, and nights spent watching metrics evolve on Tensorboard and adjusting the process.
3- The Enrichment Layer
When listings are collected, we need to enrich them with our Machine Learning system.
It's a process during which we:
- Download the images
- Send them to our Machine Learning system in charge of classification
- Perform any post-processing based on the textual content
- Calculate the listing's attractiveness (how good of a deal it is)
- Update it in the database
4- The Data Access Layer (The API)
Once listings are enriched, they're ready to be displayed on our website. For this, we need to set up an API, to make listing information available to clients. I naturally went with FastAPI (the most up-to-date Python web framework at the moment, asynchronous by nature and well-implemented).
I have few comments to make about this API, which is fairly classic overall. It retrieves data, caches it, and serves it on demand. Two parts seem particularly interesting:
Building the list of best deals:
What is a good deal? A good deal is a listing that is:
- Published recently
- Near you
- At a rock-bottom price
When a user asks to see good listings, these 3 criteria are taken into account and weighted to provide the most relevant list.
Geographical filtering:
A user on the DealTracker website has the ability to do geographical filtering, and say "what are the listings within X km radius around such and such city". When they do this filtering, it sends a request to our API, to return a filtered list of listings.
This geographical filtering of listings is done with geopandas, an enriched version of pandas dedicated to processing geographical information. After finding a list of vector geodata for each French municipality, I was able to implement the algorithm that, from the location sent by the user, finds all listings located within X km.
5- The Frontend
The goal of the Front-end is to provide an interface for users to access the results of our analysis. This interface must be:
- Fluid
- Ergonomic
- User-friendly
- Optimized for SEO
For this site, I ended up going with NextJS. It wasn't my first choice, I had initially decided to do it in Flutter web...
The design is done with antd, and I put real work into the hierarchy of relevant information. Of course, you can access the website at https://www.dealtracker.org
Data Service, Front-end
To serve data at high performance, I set up a FastAPI.
On the Front-end side, I wanted the best possible performance, so I went with NextJS (app router), with an output consisting solely of static files. I put a lot of work into the interface and SEO optimization.
This part required me to work on various topics:
- Geographical filtering with Geopandas
- API design to limit database calls
- Optimizing information serving speed at the cache level
- Defining data flows between front-end and backend
- Studying the best available front-end libraries