Instacart
Instacart uses LLMs, including a tuned Llama-3-8B, to understand rare grocery searches
Instacart replaced several older search-intent models with large language models. For rare, oddly phrased searches, a fine-tuned open-weight Llama-3-8B cut average scroll depth by 6% and complaints about those results by 50%, per Instacart's engineering blog.
Key figures
Copied exactly from the sources. Copy any figure with its citation.
6%
Reduction in average scroll depth for tail (rare) queries after the new LLM tagging, per A/B test1
50%
Reduction in user complaints about poor search results for tail queries1
over 95%, with 90%+ precision
Query rewrite coverage with the LLM approach, across three rewrite types1
50%
Query rewrite coverage of the legacy system, as a share of search traffic1
What it took
The problem
When shoppers search on Instacart, they type things like "bread no gluten". The company calls the work of understanding such queries query understanding. For years it relied on separate machine learning models for each task. Rare searches were hard for them because there was little click history to learn from. The old rewrite system also covered only about half of search traffic. Keeping many separate systems slowed development.
What they built
Instacart moved query understanding to large language models (LLMs) in three tasks. One sorts a query into the product catalog's category tree. A second rewrites queries as substitutes, broader searches or synonyms. A third tags parts of a query, such as the product, brand and attributes.
For tagging, Instacart built a two-part system. A large model works offline on common queries and stores validated answers in a cache. Rare queries go to a small model, a fine-tuned Llama-3-8B. Open-weight means the model files are published, so a company can adapt and host them.
How they did it
Instacart feeds its own data into the prompts: purchase history and catalog details. This technique is called retrieval-augmented generation. After the model answers, checks against the product catalog remove invented or off-target results. The offline model's answers also became training data for the small model, so the small one learned to copy the larger one.
Speed was the main hurdle. Out of the box, the small model took nearly 700ms on A100 GPUs. Merging the tuning layer into the base model and moving to H100 GPUs reached the 300ms target. The team tested a lower-precision setup that was about 10% faster but slightly hurt recall, and did not use it. It also added autoscaling, so fewer GPUs run at quiet hours.
Results
Instacart reports that the small model matches a much larger one, with 96.4% precision against 95.4%, slightly lower recall and similar F1. In A/B tests, the tail-query tagging cut average scroll depth by 6%, so shoppers found items faster. Instacart says the system is live, serves millions of cold-start queries weekly and cut complaints about poor tail-query results by 50%. LLM rewrites raised coverage above 95% with 90%+ precision.
What to copy
Use big models to prepare data and answers offline, then serve a small tuned model for the long tail.
All reported figures
| Measure | Figure | Citation |
|---|---|---|
| Reduction in average scroll depth for tail (rare) queries after the new LLM tagging, per A/B test | 6%1 | |
| Reduction in user complaints about poor search results for tail queries | 50%1 | |
| Query rewrite coverage with the LLM approach, across three rewrite types | over 95%, with 90%+ precision1 | |
| Query rewrite coverage of the legacy system, as a share of search traffic | 50%1 | |
| Precision of the production fine-tuned 8B model for query tagging, versus a much larger baseline model | 96.4% vs 95.4%1 | |
| Cold-start queries served by the real-time model | millions weekly1 |
Lessons for your project
Consolidate before you scale. Instacart replaced separate models for classification and rewrites with a shared LLM approach that is easier to maintain.
Spend the expensive model's effort offline. Instacart cached results for common searches and used a small tuned model only for rare ones.
Add business checks around the model. Instacart filters outputs against its product catalog and a similarity threshold to remove invented results.
Set a latency target before launch. Instacart worked out hardware and tuning changes to reach about 300ms and skipped a faster option that lowered recall.
Caveats
- All figures come from Instacart's own engineering blog. No independent source confirms them.
- The Instacart page blocked automated access, so this case was written from an archived copy of the post.
- Only the real-time tagging model is named (Llama-3-8B). The post does not name the models behind category classification or query rewrites.
- The post gives no go-live date. The deployed date shown is the post date.
- The 6% and 50% results apply to rare queries (the post cites the bottom 2% of queries), not to all searches.
- The post gives no cost or team figures.
Sources
- Building The Intent Engine: How Instacart is Revamping Query Understanding with LLMs
tech-at-instacart (Instacart Engineering) · 2025-11-13 · Company blog · tech.instacart.com
Summarized in our own words from the sources above. Spotted an error? See corrections.