François Chollet YC AI Startup School Notes

2024 marked a cultural shift in AI: from static pretraining to test-time adaptation (TTA).
- TTA allows models to change internal state and adapt on the fly.
- This led to breakthroughs on the ARC benchmark, which tests general intelligence.
Old Paradigm: Memorized, static skills—strong but rigid.
New Paradigm: Models that learn and adapt in real-time, emulating how humans generalize.

Two schools of thought:
1. Intelligence as a set of crystallized abilities (static skills).
2. Intelligence as a process of abstraction and adaptation.
Chollet argues for the second:

“Intelligence is the conversion ratio between the information you have and the goals you can achieve.”
Benchmarks like Kaggle and Netflix Prize often mislead:
- High accuracy doesn’t mean high usability or intelligence.
- Success in static benchmarks ≠ meaningful progress in AGI.

Most benchmarks measure memorization, not generalization.
- Example: Chess-playing AIs didn’t teach us much about real-world intelligence.
Chollet’s call: we need to target fluid intelligence:
- The ability to adapt, invent, and problem-solve in novel situations.
- Not just "solve known problems" but "learn to solve unknown problems".

ARC-1 (2019): A "machine IQ test" of 1,000 unique, small problems.
- Solvable by humans using general reasoning, not memorization.
- Tasks involve geometry, counting, basic logical patterns.
ARC-2 (2024):
- Maintains ARC-1 format but tasks are more complex, requiring deliberate thinking.
- Validated on 400+ humans (e.g., Uber drivers, students) to ensure solvability.
- No task solved by memorization; requires fluid intelligence.
ARC-3 (Coming 2026):
- Introduces interactive reasoning, agency, and autonomous goal-setting.
- Focus: not just thinking, but learning actively and acting.
- Will be launched with a developer preview in July 2025.