How to Balance Inference Cost and User Experience for Agents

Amplitude tested whether switching from Claude Sonnet 4.6 to Google’s Gemini 3 Flash would reduce costs without harming user experience. The cheaper model cut session costs by 52% but increased response latency by 88% and reduced user engagement by 10%, prompting the team to hold off on the switch despite conversion remaining flat. The experiment shows how Agent Analytics can measure the true impact of AI model changes on product outcomes.