IntroductionProject originsGlobal architectureObject identificationLLM in productionCounterfeit detection
← DealTracker

DealTracker - LLM in production

Why an LLM in the pipeline

The vision model decides from the images. But a listing isn't just images: the title, description and declared category carry just as much information. A vision model alone can't answer questions like "is this listing actually selling the product it claims to, or just its box?". So I added a multimodal LLM to the pipeline, in production, right where the decision to publish or reject a listing happens.

What it actually does

Each listing goes through a filtering step that combines two signals:

  • keyword rules, fast and free, which eliminate the obvious cases;
  • a multimodal LLM, which receives the listing's images and text and answers a simple question: does it match a targeted product, and is it a good deal?

The LLM is served from my homelab, by a dedicated inference server. It runs open-weights, multimodal models: it "sees" the images and reads the text, and its answer feeds directly into the pipeline's filtering step.

One rule I set after the first false negatives: photos showing only the box or packaging are NOT enough to reject a listing. If the text sells the product, even sealed in its box, the answer is yes. Only when the text explicitly sells the box alone, or when nothing shows the product, is the listing dropped.

Cost and availability

The pipeline runs continuously, with no human intervention, on hundreds of new listings per day. The inference server is monitored by availability probes like any other service: if the LLM becomes unavailable, filtering continues on keywords alone and the component recovers. Running this service at home wasn't just about budget: keeping control over latency and the confidentiality of production data mattered at least as much.

I wanted to make sure DealTracker could handle any volume. So I optimized this process by maximizing its processing throughput, and ensuring it was scalable.

Regarding processing throughput, the most important thing is to set up an asynchronous processing pipeline. Meaning that tasks are not processed sequentially, they are orchestrated by Taskiq (a Celery/Dramatiq equivalent with asyncio). The system's bottleneck is the PyTorch model, which uses 4GB of VRAM, and can process around 450 listings per minute on my RTX 3060 Ti.

Regarding the scalability of this pipeline, this is naturally enabled by taskiq, since tasks don't hold state, and are decoupled. It's possible to add hardware capacity to handle each of the tasks, which means the system is as capable as the hardware it's deployed on.

Production Inference

The inference pipeline processes listings continuously through a multi-step asynchronous architecture:

  • Image retrieval
  • GPU inference (PyTorch + TensorRT)
  • Multi-image aggregation
  • Linguistic rule filtering
  • Calculation of a listing's attractiveness relative to the median price of an object

The pipeline throughput is 450 listings per minute on an RTX 3060 Ti.