Kushagra Rastogi

Explore the mechanics of gradient synchronization in PyTorch distributed training, focusing on MPI primitives like All-Reduce and core techniques like pipeline parallelism, tensor parallelism, and sharded data parallelism.