Benchmark results

Evaluation results

Evaluation Matrix

I tried to sum up the evaluations in 2 dimensions: end user UX + data team UX.

If you want to maximize end user UX + data team UX you have to go for AI native BI tools. But it means you need to 1/ migrate your existing BI, 2/ pay a lot.
Existing BI tools will maximize end user UX - they already know the UX - but their AI features are not really mature.
Generalist AI tools also are good end user UX - they already use them - but harder to customize by the data tool.
Warehouses have a good data team UX, but still are hard to put in hands of business users - and not every warehouse has the same AI maturity.
AI editors are working good for data teams, but are not ready for business users.
Text to SQL tools seem still WIP - end user UX is new, data teams often need to rebuild context in them, and have no guarantee of reliability.

Main scoring

	End user UX		Reliability				Speed	Cost	Data team UX
	Interactive charts	Transparent loop	Use case answer
is right	Evals	Logs	Context options	Time to answer	Total cost	Vendor lock-in	Set up ease
Warehouses
Snowflake Cortex	✅	❌	🔴	❌	❌	🟠 no external semantics	🟢 15s	🟠 snowflake cost - no LLM cost	🔴 semantics built-in	🟢
Databricks Genie	✅	✅	🟢	✅	✅	🟠 no external semantics	🟢 30s	🟠 databricks cost - no LLM cost	🔴 semantics built-in	🟢
Clickhouse Librechat	❌	✅	🔴	❌	❌	🟠 mostly via MCPs	🔴 50s	🟢 Open Source - only LLM costs	🟢	🔴
BI tools
Looker	✅	❔	❔	❔	❔	❔	❔	🔴 Looker cost - expensive	🔴 semantics built-in	🟢
Metabase	✅	✅❔	🔴	❌	✅	🔴 no context option	🟠 40s	🟠 $200/month + $6/seat	🔴 semantics built-in	🟢
Omni	✅	✅	❔	❌	✅	🟢	❔	🔴 Omni cost - min $40K/year	🟠 AI rules built-in	🟢
Lightdash	✅	❌	🔴	✅	✅	🔴 prompt only	🟠 40s	🔴 $3000/month	🟢	🟢
Notebooks
Hex	✅	✅	🔴	❌	✅	🟠 data + semantics only	🔴 1min	🔴 $75/seat	🟠 semantics to import	🟠
Count	✅	✅	🔴	❌	❌	🟠 data + semantics only	🔴 50s	🟠 $49/seat for small orgs	🟠 semantics in count	🟢
General agents
Claude	✅	✅	🔴	❌	❌	🟢	🟢 30s	🟢 $17/seat	🟢	🔴
Claude Cowork	✅ (ugly ones)	✅	🔴	❌	❌	🟢	🔴 1min	🟢 $17/seat	🟢	🔴
Dust	✅	✅	🔴	❌	✅	🟠 mostly via MCPs	🟢 30s	🟠 $35/seat	🟠 connections built-in	🟢
Codex	✅ (ugly ones)	✅	🟢	❌	❌	🟢	🔴 1min	🟢 $20/seat	🟢	🔴
Text to SQL tools
Dagster Compass	❌	✅	🔴	❌	✅	🔴 metadata only	🟢 35s	🟠 $0-$500/month	🟢	🟢
Cube	✅	✅	🔴	❌	✅	🟠 no external semantics	🟠 40s	🟢 $40/seat	🟠 reusable semantics	🟠
TextQL	❌	✅	🔴	❌	❌	🟠 no external semantics	🔴 1min	🟠 usage based - $0-$100/seat	🔴 proprietary ontology	🔴
Julius	❌ matplotlib	✅	❔	❌	❌	🟠 mostly via MCPs	❔	🟠 $375/month	⚪️ no semantics	🟢
Fabi.ai	✅	✅	🔴	❌	❌	🔴 metadata + prompt	🟢 25s	🟠 $39/seat	⚪️ no semantics	🟢
Thoughtspot	✅	✅	🔴	❌	❌	🟠 no external semantics	🟠 40s	🟠 $50/seat	🔴 proprietary model	🔴
IDE
nao	✅	✅	🟢	❌	❌	🟢	🟢 15s	🟢 $30/seat	🟢	🟢
Cursor + MCP	❌	✅	🟢	❌	❌	🟢	🟢 30s	🟢 $20/seat	🟢	🟠

Available context by solution

	Tables selection	Tables metadata	Data sampling	dbt repo	dbt docs	Prompt	Rules	Skills	Tool semantic	Any semantic	Queries	Joins	Dashboards	Model choice	MCPs
Warehouses
Snowflake Cortex	✅	✅	✅	❌	❌ (table desc)	✅	❌	❌	✅	❌	✅	✅	✅	❌	❌
Databricks Genie	✅	✅	❌	❌	❌ (table desc)	✅	❌	❌	❌	❌	✅	✅	❌	❌	❌
Clickhouse Librechat	✅ via MCP permissions	❌	❌	✅ via MCP	✅ via MCP	✅	✅	❌	❌	✅ via MCP	❌	❌	❌	✅	✅
BI tools
Looker	❔	❔	❔	❔	❔	❔	❔	❌	❔	❔	❔	❔	❔	❌	❔
Metabase	✅	✅	❌	❌	❌	❌	❌	❌	✅	❌	✅	❌	✅	❌	❌
Omni	✅	✅	❌	✅	✅	✅	✅	❌	✅	✅ dbt	❌	❌	❌	❌	❌
Lightdash	❌	✅	❌	✅	✅	✅	❌	❌	✅	✅ dbt	❌	❌	❌	❌	❌
Notebooks
Hex	✅	✅	❌	✅	✅	✅	❌	❌	✅	✅ dbt	❌	❌	❌	❌	❌
Count	❌	✅	❌	✅	✅	✅	❌	❌	✅	❌	❌	❌	❌	❌	❌
General agents
Claude	✅ via MCP permissions	❌	❌	✅ via MCP	✅ via MCP	✅	✅	✅	⚪️	✅ via MCP	✅	✅	✅ via MCP	✅	✅
Claude Cowork	✅ via MCP permissions	❌	❌	✅ local folder	✅ local folder	✅	✅	✅	⚪️	✅ via MCP	✅	✅	✅ via MCP	✅	✅
Dust	✅ via MCP permissions	❌	❌	✅ via MCP	✅ via MCP	✅	❌	✅	⚪️	✅ via MCP	❌	❌	❌	✅	✅
Codex	✅ via MCP permissions	❌	❌	✅ local folder	✅ local folder	✅	✅	✅	⚪️	✅ via MCP	✅	✅	✅ via MCP	✅	✅
Text to SQL tools
Dagster Compass	✅	✅	✅	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌
Cube	✅	✅	❌	❌	❌	✅	❌	❌	✅	❌	✅	❌	❌	❌	❌
TextQL	✅	✅	❓	❓	❓	✅	❌	❌	✅	✅ via MCP	❌	❌	❌	❌	✅
Julius	❌	✅	❌	✅ via MCP	✅ via MCP	✅	❌	❌	❌	✅ via MCP	❌	❌	❌	❌	✅
Fabi.ai	✅	✅	❌	❌ cloud only	❌	✅	❌	❌	❌	❌	❌	❌	❌	❌	❌
Thoughtspot	✅	✅	❌	✅	✅	✅	✅	❌	✅	❌	✅	✅	❌	❌	❌
IDE
nao	✅	✅	✅	✅	✅	✅	✅	✅		✅ via MCP	✅	✅	✅ via MCP	✅	✅
Cursor + MCP	✅ via MCP permissions	✅ via MCP	✅ via MCP	✅	✅	✅	✅	✅		✅ via MCP	✅	✅	✅ via MCP	✅	✅