Concepts
Explainer
Selected: nothing
What & why
Training = minimizing a loss. Gradient descent does it by repeatedly stepping downhill: compute the gradient (the direction of steepest increase) and move the opposite way. Optimizers like Momentum and Adam change HOW you step to converge faster and avoid getting stuck or zig-zagging.
Analogy
A ball rolling downhill in fog: you can only feel the local slope (gradient). SGD takes fixed small steps; Momentum keeps rolling so it doesn’t stall; Adam shortens steps where it’s steep and lengthens them where it’s flat.
Used in
- Training every neural network (SGD, Momentum, Adam/AdamW)
- Optimizers in PyTorch / TensorFlow / JAX