Shopify
Shopify replaces a frontier model with a fine-tuned Qwen3-32B agent in Sidekick Flow
Shopify trained its own agent on the open-weight Qwen3-32B model to turn merchants' plain-English requests into Shopify Flow automations inside its Sidekick assistant. Shopify says it is 2.2x faster and 68% cheaper than the frontier model it replaced, and now serves most production traffic.
Key figures
Copied exactly from the sources. Copy any figure with its citation.
2.2
x faster
Speed of the fine-tuned Flow agent compared with the frontier model it replaced (baseline model and measurement method not stated)1
68%
cheaper
Cost of the fine-tuned Flow agent compared with the frontier model it replaced (baseline and method not stated)1
35%
lower
Workflow activation rate of the fine-tuned model versus the prompt-based agent in the first test on 1% of traffic, before later improvements1
What it took
The problem
Shopify Flow lets store owners automate tasks, such as tagging orders or sending emails, by building workflows from triggers, conditions and actions. Most store owners are not engineers, and a blank canvas is daunting. Shopify's AI assistant, Sidekick, writes the workflow from a plain-English request. Shopify ran this on a frontier model it did not control, and wanted a faster, cheaper option it could improve on its own data.
What they built
Shopify fine-tuned Qwen3-32B, an open-weight model, meaning its files are published so a company can adapt and run them. The result is an agent that calls tools to search for Flow building blocks, configure them and assemble the workflow. Shopify reports it runs 2.2x faster at 68% lower cost than the frontier model it replaced, and says it beats closed models. It now handles most of the production traffic for this skill.
How they did it
There were no real conversations to learn from before launch. The team started from existing merchant workflows that had actually run, and used a stronger model to write a plausible request for each. Shopify says building the step-by-step tool-call sequences for each example was the bulk of the engineering effort.
Flow's native format is hard for a model to write reliably. Shopify showed the model Python instead and converted it back with a translator that was tested on every workflow built through Sidekick. On the same training data, writing Python instead of the native format lifted the share of workflows free of errors by 22 points and the share doing what was asked by 13 points.
Training runs on two nodes of H200 graphics processors. A full run takes 12 hours, which makes weekly retraining practical. Offline tests reached parity with the old agent, but the first live test on 1% of traffic showed a 35% lower activation rate. The gap came from requests the training data lacked, such as editing existing workflows and email setup. Shopify then set up a weekly loop: score production conversations with a calibrated AI judge, add the good ones to training, find gaps, retrain and redeploy.
Results
Shopify reports the fine-tuned agent is 2.2x faster and 68% cheaper than the frontier model it replaced, and that it now serves most production traffic. It says the improvement loop closed the early activation gap in two weeks. The same approach is now being applied to other Sidekick skills.
What to copy
Pick one well-defined task and prove it on live traffic at a small share before switching over. Plan for a continuous cycle of measurement and retraining, not a one-off build.
Lessons for your project
Test on real traffic early. The model matched the offline benchmark, yet a 1% live test showed an activation rate 35% lower, because real merchants asked things the test set missed.
Start with a narrow skill that has a clear tool-calling task. Shopify isolated one skill, Flow generation, rather than replacing the whole assistant.
Build a feedback loop before you need it. Shopify scores production conversations with a calibrated AI judge and retrains weekly.
Do not rely on one business metric alone. Shopify found activation rate reflects merchant behavior as well as model quality, and added an expert-calibrated judge.
Caveats
- All figures come from Shopify's own engineering post. No independent source confirms them.
- The post does not name the frontier model it replaced and does not explain how speed and cost were measured, so '2.2x faster' and '68% cheaper' cannot be checked.
- The post page prints only 'Updated by colleagues on Apr 22, 2026', not a separate publication date, so that date is used as the source date and as the deployed date (the announcement date). The post says the system ran on a frontier model 'six months ago', so the actual go-live was earlier and is not stated.
- The 35% lower activation rate was the first result at 1% of traffic. The post says the improvement loop closed the gap in two weeks, but gives no final activation figure.
- 'The majority of production traffic' is not quantified.
- Training cost and team size are not stated.
Sources
- Flow generation through natural language: An agentic modeling approach
Shopify Engineering · 2026-04-22 · Company blog · shopify.engineering
Summarized in our own words from the sources above. Spotted an error? See corrections.