Backpropagation
Definition
The fundamental algorithm for training neural networks, which computes the gradient of the loss function with respect to each weight by propagating errors backward through the network.
Backpropagation (backprop) is the mathematical engine that makes neural network training possible. After a forward pass produces a prediction and a loss is calculated, backpropagation uses the chain rule of calculus to compute how much each weight in the network contributed to the error. These gradients are then used by an optimizer (like gradient descent) to update the weights in the direction that reduces the loss. The algorithm was popularized for neural networks by Rumelhart, Hinton, and Williams in 1986. Despite decades of research into alternatives, backpropagation remains the primary method for training all modern neural networks, from small classifiers to trillion-parameter language models. Efficient implementation of backprop on GPUs through frameworks like PyTorch and TensorFlow is essential for modern deep learning.
Related Terms
Deep Learning
A subset of machine learning that uses multi-layered neural networks to learn hierarchical represent...
Neural Network
A computing system inspired by biological neural networks, consisting of interconnected nodes (neuro...
Loss Function
A mathematical function that measures how far a model's predictions are from the actual target value...
Gradient Descent
An optimization algorithm that iteratively adjusts model parameters in the direction that most reduc...