Distributed training spreads the computational work of training an AI model across multiple machines or processors simultaneously, rather than training on a single device. Given that today's largest models can have hundreds of billions of parameters, training them on one machine would take an impractically long time, if it were even possible given memory constraints.
There are a few common strategies for splitting the work: data parallelism, where each machine trains on a different slice of the training data using a copy of the same model, and model parallelism, where different machines each hold and process a different part of the model itself. Distributed training is what makes it computationally feasible to train models at the scale of GPT-4 and similar large language models within a reasonable timeframe, turning a task that would take years on a single machine into one that can run in weeks across thousands of coordinated processors.
