Day 1 Resources:

• Primer: https://gpu-primer-vizuara.vercel.app/
  • Homework to read through the complete primer to be better prepared for Day 2
• The Ultra-Scale Playbook:Training LLMs on GPU Clusters
  • This is used as reference for the cohort
• Case Study: Unsloth Gradient Checkpointing - 4x longer context windows
• Arxiv papers for more case studies:
  • GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection
  • DeepSeek-V3 Technical Report
• LLM architecture:
  • https://www.youtube.com/playlist?list=PLPTV0NXA_ZSgsLAr8YCgCwhPIJNNtexWu
  • A Bird’s-Eye View of LLM Architecture
• Additional resources shared by other participants:
  • How to Scale Your Model

Day 2:

Jupyter/Google Colab Notebooks:

• https://drive.google.com/file/d/1uF6qRVRVpick0JZ2GAnRK4i3W2tC8NF3/view
• Llama Memory Track: https://drive.google.com/file/d/1YyQsqbJT5aCuG9sfBn7TvtOegQNcricC/view?usp=sharing

Extra reading:

• Adam optimizer → we store optimizers in memory (GPR VRAM), so it's good to learn about the Adam optimizer used in LLMs.
  • https://vizuaraai.github.io/adam-visualizer/
  • https://arxiv.org/pdf/1412.6980
• Optimizer used in Kimi models: https://kellerjordan.github.io/posts/muon/
• How LLMs Scaled from 512 to 2M Context: A Technical Deep Dive

Day 3:

• Activation recomputation: https://vizuaraai.github.io/shallowspeed-visual-guide/activation-recomputation.html

• Google Colab:

  • Actiovation recomputation: https://drive.google.com/file/d/1n2F1GjjuoVODw9--lzxPPsIjJlZdI2X4/view?usp=sharingActivation recomputation: https://drive.google.com/file/d/1n2F1GjjuoVODw9--lzxPPsIjJlZdI2X4/view?usp=sharing
  • Shallow speed:

  https://drive.google.com/file/d/1aKXRKdcA2kXD7phXVOlXyK8sPWpTJD6h/view

• https://siboehm.com/articles/22/data-parallel-training