The Impact of Learning Rate Choice

The learning rate α has a huge impact on gradient descent performance. A poor choice can make gradient descent extremely slow or fail entirely.

Case 1: Learning Rate Too Small

What happens:

• Each update step is tiny (e.g., α = 0.0000001)
• Cost J decreases, but very slowly
• Takes many, many steps to reach the minimum

Visualization:

Start → tiny step → tiny step → tiny step → ... → (eventually) minimum

Result: Gradient descent works, but is painfully slow.

Case 2: Learning Rate Too Large

What happens:

• Update steps are huge
• May overshoot the minimum entirely
• Can land at a point with higher cost than before
• Each subsequent step may overshoot further

Visualization:

Start (near minimum) → overshoot right → overshoot left → overshoot further right → ...

Result: Gradient descent fails to converge and may diverge (get worse and worse).

Summary: Learning Rate Effects