Kasisto's fine-tuned Gemma 3 model answers banking questions at 40+ institutions

The problem

Banks want virtual assistants that answer customers' questions from current rate sheets and policies. General-purpose AI models can make up answers, are slow and cost money on every query. In a regulated business a wrong answer about, say, an early mortgage payoff penalty can create legal and financial risk. The assistant must also decline to answer when the documents do not contain the answer.

What they built

Kasisto, a company that supplies virtual assistants to banks, built FinRAG-12B. It is Google's open-weight Gemma 3 12B model, fine-tuned to read passages a bank's search system retrieves, write an answer from them, cite where each claim came from and refuse when there is no support. This pattern is called retrieval-augmented generation (RAG).

The model is part of the assistants Kasisto runs for banks. The paper says it is deployed at more than 40 financial institutions. At the credit union it was compared with a legacy production model; the paper separately names an earlier Kasisto model, FinRAG-v3, built on Mistral-7B-Instruct.

How they did it

Kasisto trained the model in two stages. The first used public financial data to teach answer formats and citations. The second used Kasisto's own banking conversations to teach when to refuse. About 22% of the training examples were questions with no answer in the documents. The team filtered training data with an AI judge to keep quality high and used only 143 million tokens.

The authors give the training bill as about $1,800 of GPU time. For production the model is compressed to 4-bit weights, which shrinks it from 24 GB to 8.4 GB and lets it run on a single GPU.

Results

At one large US credit union, the authors compared a legacy production model with FinRAG-12B on 3,297 sampled queries over seven months. The share of queries resolved without handing off to a human rose from 77.4% to 84.5%, a gain of 7.1 percentage points. In the authors' offline latency tests, which are not production measurements, the new model was also 3-5x faster than GPT-4.1, about 20-50x cheaper per query on published API prices, and took 3.2x less time to finish a response than FinRAG-v3.

In offline tests on 258 banking questions, the model refused 12% of questions, between the base model's 4.3% (too few refusals) and GPT-4.1's 20.2% (too many).

What went wrong

Satisfaction scores moved less than resolution. Overall satisfaction rose from 59.5% to 62.9%, but the paper says the difference could be chance. Satisfaction for resolved and unresolved queries stayed nearly flat, so the authors conclude that fewer unresolved queries drove the improvement. The tuned model also cited fewer passages than the untuned base model, a trade-off the authors accept for more selective citations.

What to copy

Fine-tune a small open-weight model on your own support material only after setting a clear target for refusals. Run it beside your current model in the same pipeline and compare resolution rates on live queries. Work out per-query cost before choosing between self-hosting and a pay-per-use API.