📄 Summary: This paper identifies a critical bottleneck in extending language models with new vocabulary tokens: mean initialization collapses tokens into a degenerate subspace, erasing distinctions that fine-tuning cannot fully recover. The authors use spectral and geometric analysis to diagnose this problem and propose grounded initialization strategies to better preserve inter-token distinctions for domain-specific tasks like semantic recommendation.
💡 Key Insight: Simply averaging existing embeddings when adding new tokens to language models backfires by destroying the uniqueness you're trying to create.
📄 Summary: This paper introduces a minimalist training approach where language models solve multiple problems simultaneously in a shared context window, creating an implicit token budget that forces efficient reasoning. The method discovers a novel task-scaling law showing how concurrent problem solving trades off quality and efficiency without complex training pipelines.
💡 Key Insight: Teaching models to juggle multiple problems at once naturally teaches them to reason more efficiently without explicit penalties.
📄 Summary: The paper demonstrates that no single LLM excels at generating diverse responses across all prompts, but introduces a "diversity coverage" metric to measure answer set quality and proposes a router that learns to select the best model per-prompt for comprehensive response generation.
💡 Key Insight: Different models have hidden strengths for different types of open-ended questions—smart routing beats picking one winner.
📄 Summary: De Jure is a fully automated pipeline for converting dense legal documents into machine-readable regulatory rules through LLM-driven decomposition, multi-criteria evaluation, and iterative repair, requiring no human annotation or domain expertise. The system normalizes documents, structures rules semantically, judges quality across 19 dimensions, and automatically fixes low-scoring extractions within a bounded budget.
💡 Key Insight: Legal compliance automation is possible by letting LLMs iteratively fix their own rule extraction errors without human lawyers.