By Chaminda Bandara (wgcban.com)

Setting up Distributed Data Parallel

Based on the following repos:

Video series: https://youtu.be/-K3bZYHYHEA?si=6oJA65LybhwDYotp

Github: https://github.com/pytorch/examples/tree/main/distributed/ddp-tutorial-series

PyTorch Tutorial: https://pytorch.org/tutorials/beginner/ddp_series_theory.html

Informative lecture: https://youtu.be/TibQO_xv1zc?si=3daCXI9m5bhSC5X1

Data Parallel

model = MyModel()
model = nn.DataParallel(model)

Identical copies of the model, different sub-batches, synchronized updates

• loss values l_1 and l_2 are different. Hence, needs to synchronize the updates. Some communication happens.

• Synchronization is important, otherwise we will converge to two different solutions.
• At each iteration of data parallel,
  • Replicate model to all GPUs
  • Split the batch to all GPUs
  • Sync the updates (uses thread, not independent processes)

Why DDP?

Multi GPU Training with DDP

0. Introduction