Faire
Faire fine-tunes Llama 3 8B to score search relevance, beating its tuned GPT model
Faire, a wholesale marketplace, fine-tuned Meta's open-weight Llama3-8b to judge whether search results match what retailers type. Faire reports a 28% gain over its earlier fine-tuned GPT model, and the model, hosted on its own GPUs, reached 70 million predictions a day when Faire backfilled labels at scale.
Key figures
Copied exactly from the sources. Copy any figure with its citation.
28%
Gain in Krippendorff's Alpha (agreement with human labelers) for the best fine-tuned Llama3-8b versus Faire's existing fine-tuned GPT model, in relative terms1
70 million
using 16 GPUs
Relevance predictions per day reached when backfilling labels at scale on a self-hosted GPU cluster1
0.56
Krippendorff's Alpha of the earlier fine-tuned GPT model when it went to production1
What it took
The problem
Faire is an online wholesale marketplace. Retailers search it to find products for their shops. When results do not match the query, shoppers lose trust in the platform. Faire needed to measure how relevant its search results were. At first it paid an outside vendor to label a sample of search results each month by hand. That was costly, and the labels arrived a month late.
What they built
Faire built a model that reads a search query and a product listing and rates the match on a four-step scale: exact, substitute, complement or irrelevant. A first version used a fine-tuned GPT model. Fine-tuning means training an existing model further on your own labeled examples. That version made daily measurement possible but was limited by the per-use fees of the API.
Faire then fine-tuned open-weight Llama models from Meta. Open-weight means the model files are published, so a company can run them on its own hardware. The best result came from Llama3-8b.
How they did it
The team first wrote labeling rules and reached more than 90% agreement among human labelers. It trained Llama2-7b, Llama2-13b and Llama3-8b on three sizes of labeled data and compared them with the GPT model on about 5,000 held-out records. It trained on 8 A100 GPUs using LoRA, a method that updates only a small share of a model's settings, here about 4%. Training used GPUs the company already owned for other deep learning work.
For daily use, the model runs on Faire's own GPU cluster. The team compressed it to 8-bit numbers, ran batches on single A100 cards and added more machines as needed.
Results
Faire reports a 28% gain in agreement with human labels over its production GPT model. The larger the labeled dataset, the better the results, and the gap between base models shrank as data grew. When Faire filled in labels for past searches, the cluster reached 70 million predictions a day on 16 GPUs. Faire says this lets it measure search relevance daily and analyze personalization and experiments.
What to copy
Define quality in terms people agree on, build labeled data, then test small open models against the paid model you use today.
Lessons for your project
Write a clear definition of what good looks like before using a model. Faire adopted a four-tier relevance scale and refined labeling rules until human labelers agreed more than 90% of the time.
Invest in labeled examples first. Faire found the volume, quality and mix of labeled data mattered most, more than the choice of base model.
Compare a small open model with the paid API you already use. Faire's tests showed a tuned 8B-parameter model could beat its tuned GPT model at lower serving cost.
Start with an internal use. Faire used the model to measure search quality offline before considering any change to what shoppers see.
Caveats
- Faire reports all figures in its own engineering blog. No independent source confirms them.
- The Faire page blocked automated access, so this case was written from an archived copy of the post.
- The post does not give a go-live date. The deployed date shown is the post date.
- The model is used offline, to measure how relevant search results are. The post says real-time use is a future step, so shoppers do not yet see results changed by it.
- Faire says it lowered cost compared with outside LLM providers but gives no dollar figures, and does not name the GPT model or provider.
- Faire states that improvements are in relative terms unless otherwise specified.
Sources
- Fine-tuning Llama3 to measure semantic relevance in search
The Craft (Faire engineering blog) · 2024-04-29 · Company blog · craft.faire.com
Summarized in our own words from the sources above. Spotted an error? See corrections.