What is Learning Rate? — AI Encyclopedia | XLUXX

Learning Rate — How big of a step the model takes when updating weights during training. Too high and the model overshoots optimal values. Too low and training takes forever. Learning rate scheduling (warmup, cosine decay) is standard practice. The single most important hyperparameter.

Part of the XLUXX AI Encyclopedia — the most comprehensive AI reference on the web.