Masked language modeling is a training technique in which certain words in a sentence are deliberately hidden, or "masked," and the model is trained to predict what those hidden words are based on the surrounding, unmasked context. Because the model can see words on both sides of the mask, it learns to understand a word's meaning from its full context, not just from what came before it.
This bidirectional approach is what distinguishes masked language modeling from the autoregressive, left-to-right prediction used by models like GPT, which only ever see the words that came before the one they're currently predicting. BERT (Bidirectional Encoder Representations from Transformers) is the best-known example of a model trained this way, and masked language modeling remains a common pretraining technique for models built primarily for language understanding tasks like classification and search, rather than open-ended text generation.
