****Every major AI lab is racing to build longer context windows. A million tokens, ten million, unlimited. The assumption buried underneath all of that is: if you can just fit everything in, the memory problem is solved. I think that's the wrong bet entirely. I think memory isn't a capacity problem. It's an intelligence problem. And almost nobody is seriously working on that. This is the story of how we ended up building something we didn't plan to build, in a hackathon we almost didn't show up to.
****Our team lead's grandmother was in the hospital the week the problem statement dropped. I was deep in end-term exam prep. By the time we finally sat down and looked at what everyone else was building, most teams were already in their training runs. We hadn't written a single line of code. The orientation is where something shifted for me. The framing was simple: build an environment that matters. Not a game, not a toy, something where the problem is real and the solution, if it works, is actually useful. And I remember sitting there thinking, okay, what problem do I actually lose sleep over?
I work on agentic systems. Specifically on memory. I've tried everything, text-based memory, vector embeddings, graph RAG, hybrid approaches. They all work. They all solve a problem. But none of them feel intelligent. They're fast rules dressed up as cognition. The most sophisticated is vector similarity search, and even that has well-documented cracks: the Q-RAG paper from November 2025 lists them flatly, static knowledge, computational inefficiency on long contexts, degraded performance from attention dilution, and hallucinations that emerge when the retrieval space gets noisy [arxiv.org/html/2511.07328v1]. I've hit every one of those in production.
There's a line I keep coming back to, from a peer-reviewed study: "The gap between 'has memory' and 'does not have memory' is often larger than the gap between different LLM backbones." Swapping your model matters less than fixing your memory. That's the whole pitch for why this deserves a serious attempt.
So I had the problem. What I didn't have yet was a way to model it as something trainable. The idea that unlocked everything
I'm an artist by instinct, I always look at how nature solves a problem before I try to engineer one. And the answer came from somewhere unexpected: the way I used to study for JEE. When you're preparing for competitive exams, you don't memorize everything. You can't, and the ones who try usually burn out. What you learn to do, if you're studying well, is remember selectively. I used to relate the periodic table to a song so the pattern would stick. I'd connect a difficult chemical name to a word I already knew, something familiar, so retrieval wasn't starting from scratch every time. And the most important thing: I wasn't storing information randomly. I was storing what kept appearing in mock tests, what I knew would be asked. My memory was being shaped by the query distribution.
The moment I saw it that way, the environmental design became obvious. What if we train an agent to do exactly this, see a stream of facts, more than it can retain, and learn not just what to store but how to phrase what it stores so it can find it again? Not verbatim storage. Not a FIFO queue. A learned anchor, a short retrieval key, freely authored, that the agent discovers is useful for bridging what it saw at storage time to what gets asked at query time.
This is the part that makes the problem genuinely hard and genuinely interesting. The facts arrive using one vocabulary, abbreviations, specific values, and technical shorthand. The queries arrive using a completely different vocabulary, full names, categorical descriptions, and natural language. "LR=3e-4" in storage versus "what learning rate was used" in the query. "grad norm 47" in storage versus "gradient instability" in the query. Vector similarity alone doesn't reliably close that gap. The anchor has to close it. And the only way an anchor gets good at closing it is through training on what queries actually arrive, which is exactly what GRPO gives us.
There's actually published research showing that when models are trained end-to-end on retrieval tasks, they can develop internal representations for their own use that are not human-interpretable in any conventional sense, optimized for machine retrieval rather than semantic legibility (Sastre & Pascual, arXiv:2506.15001). That's the direction we're pointing. The agent may find anchor strategies that look strange to a human but work precisely because they encode what a future query will search for.
****I want to be honest: building a working RL environment is genuinely hard. Getting a reward signal clean enough for GRPO to learn from, designing a curriculum that doesn't collapse in the first 50 steps, keeping the environment stable when eight simultaneous WebSocket connections are hitting it in parallel, all of this is unglamorous engineering that fails in quiet, confusing ways.
The core setup: a Qwen2.5-3B-Instruct model with LoRA adapters, trained via GRPO using TRL. The environment is OpenEnv-compliant, deployed on HuggingFace Spaces. In each episode, the agent sees a stream of facts, up to 120 at higher difficulty levels, and has a fixed memory budget far smaller than the stream. It must decide in a single pass what to store and author an anchor for each stored fact. Then queries arrive. It retrieves from its own memory and answers. The reward is binary: did the agent beat a FIFO baseline on the same seeded episode? No partial credit in the main training signal. GRPO works by ranking eight completions within a group. Clean binary outcomes produce sharp gradients, dense shaping flattens variance, and weakens the learning signal. We learned this the hard way before locking the reward design.
The curriculum is structured across five difficulty levels, each introducing exactly one new cognitive skill, so the gradient stays clean and the policy can climb without destroying what it has already learned. Level one teaches the agent just to output valid structured decisions, action grammar. Level two introduces budget pressure and distractor facts; about 30% of the stream is noise that the agent must learn to skip. Level three is where the real challenge begins: tags are removed, lexical mismatch is fully active, and the agent must learn to write anchors that genuinely bridge storage-time text to query-time text. Levels four and five push further, permanence decisions, contradiction tracking, and adversarial signals.
The reward at inference is compared against a FIFO baseline pre-computed at episode reset on the same seed, so the binary signal is always asking: "Did you do better than the dumbest possible policy on this exact problem?" That's a hard benchmark to game.
I ran the training script overnight. Went to sleep not knowing if I'd wake up to a flat loss curve or something real. My teammate sent me the graph at midnight. There was improvement, clear, unambiguous, reward-going-up improvement. I know what it takes to get a working RL environment, and seeing that signal for the first time felt genuinely good.