BLOOM (BigScience Language Open-science Open-access Multilingual): the BigScience 176 billion parameters model is currently training.

The training started on March 11, 2022 11:42am PST and will last 3-4 months on the 416 A100 GPUs of the Jean Zay public supercomputer

Follow the training at https://twitter.com/BigScienceLLM And the tensorboard, detailed model card and intermediate checkpoints at https://hf.co/bigscience/bloom Send questions about the training to bigscience-large-model-training [AT] googlegroups [.] com

More information on the BigScience project: start here BigScience

Summary:

  1. The model:
  2. The dataset:
  3. The engineering side:
  4. Environmental considerations