Objectives
The main objective is: anyone at nao can ask a question on company data without relying on the data teams.
The constraints are: it needs to be reliable, it needs to be fast, and needs to be a reasonable price.
Criterion
I defined the KPIs I want to measure, and the features I want to find to be able to measure or improve these KPIs.
There are 5 main criterion:
End user UX: How easy it is for any user - technical or not - to run analytics. The UI must be good looking, explain the agent reasoning, and give some interactive features for deep-diving.
Reliability: We want to answer as many questions as possible, in a reliable way. I don’t think this will come out of the box of the solution so we need: to be able to customize context, to eval agent on a set of unit tests, and to monitor real users usage.
Speed: The agent should be as fast as possible.
Cost: I want to monitor global cost: cost of solution + LLM cost + compute cost.
Data team UX: How easy is it to set up the tool? Does it rely on what we already have? Are we locked in the solution?
Testing a real use case
On each solution, I will test a real use case on real nao data:
What's the percentage of users who churned last month?
Why this is tricky: