Checkr
Checkr fine-tunes Llama 3 to sort its hardest background-check records, beating GPT-4 in its tests
Checkr, a background check company, fine-tuned Meta's open-weight Llama-3-8B-Instruct to classify the hardest 2% of its background check records. Checkr's engineer reports 90% accuracy on those cases, responses 30x faster than its GPT-4 tests, and inference costs 5x lower than GPT-4.
Key figures
Copied exactly from the sources. Copy any figure with its citation.
90%
Accuracy of fine-tuned Llama-3-8B-Instruct on the hardest 2% of cases1
0.15
seconds, which Checkr calls 30x faster than its GPT-4 experiments
Response time on production traffic1
What it took
Cost, team size and timeline were not disclosed in the sources.
The problem
Checkr delivers background checks for over 100,000 customers and runs millions of them each month. After a check, an employer must decide if the candidate suits the job, a step called adjudication. Checkr offers automated adjudication. For 98% of records a tuned statistical model works well. The remaining 2% are messy text records that must be sorted into 230 categories. Checkr's older neural network classified only 1% of all records with decent accuracy, leaving the other 1% unclassified and creating more work for customers.
What they built
Checkr built a classifier on Llama-3-8B-Instruct, an open-weight model from Meta. Open-weight means the model's files are published, so a company can adapt and run it. The team used fine-tuning, which means further training a model on a company's own labelled examples. It trained a small model on 150,000 rows of Checkr data.
How they did it
Checkr's machine learning engineer compared several approaches on the same data. GPT-4 given instructions and the 230 categories scored 87-88% on the 98% easier cases and 80-82% on the hardest 2%. Adding retrieval of examples lifted accuracy on the easier set but not the hard set. A fine-tuned Llama-2-7b did better than GPT-4 on both. The best result came from fine-tuning Llama-3-8B-Instruct.
The team tested several tuning and serving platforms and chose Predibase. It uses LoRAX, an open-source tool that serves many small fine-tuned add-ons (LoRA adapters) on a single base model, so new uses do not need more GPUs.
Results
The engineer reports 90% accuracy on the hardest 2% of cases, beating GPT-4 and all other fine-tuning experiments. Production responses take 0.15 seconds, 30x faster than the GPT-4 experiments. Inference costs fell 5x compared with GPT-4. The synchronous task has strict speed limits because customers expect near real-time results.
What to copy
Narrow the problem first, then compare a small tuned model with a general one on your own data before choosing.
All reported figures
| Measure | Figure | Citation |
|---|---|---|
| Accuracy of fine-tuned Llama-3-8B-Instruct on the hardest 2% of cases | 90%1 | |
| GPT-4 accuracy on the hardest 2% of cases in Checkr's experiment | 80-82%1 | |
| Response time on production traffic | 0.15 seconds, which Checkr calls 30x faster than its GPT-4 experiments1 | |
| Inference cost compared with GPT-4 | 5x reduction1 | |
| Share of all records the earlier neural network classified with decent accuracy, out of the hard 2% | 1% (the other 1% stayed unclassified)1 |
Lessons for your project
Route only the hard cases to the language model. Checkr handles 98% of records with a simple statistical model and uses the language model on the remaining 2%.
Test a small tuned model against the big general one on your own data. In Checkr's test, the tuned small model beat GPT-4 on the hard cases.
Keep prompts short once a model is tuned. Checkr found prompt changes had minimal impact on accuracy, so shorter prompts cut token costs.
Choose a serving setup that can host several tuned variants on one model. Checkr plans to add more uses without more GPUs.
Caveats
- The source is a blog post on the website of Predibase, the vendor Checkr uses, written by a Checkr engineer. No independent source confirms the figures.
- The original page now redirects to another site and blocked automated access, so this report was written from an archived copy of the original post.
- The post does not give the go-live date, so the deployed date shown is the post date.
- The post does not state where Checkr operates. The region shown is based on Checkr being a US company, not on the source.
- The post's table shows GPT-4 and tuned-Llama costs in dollars without stating the period or units, so no dollar figures are used here.
Sources
- Fine-Tuned SLMs Help Checkr Optimize Background Checks
Predibase · 2024-10-03 · Vendor case study · predibase.com
Summarized in our own words from the sources above. Spotted an error? See corrections.