1. Gradient Stability During Backpropagation

During backpropagation, gradients flow backward through the network by multiplying through each layer's weights. Consider a simple 3-layer network:

Gradient flow: ∂Loss/∂w₁ involves multiplying w₃ × w₂ × ...

With large weights (say w = 10 in each layer):

• Gradient magnitude: 10 × 10 × 10 = 1000
• The gradient explodes exponentially with depth

With small weights (say w = 0.5 in each layer):

• Gradient magnitude: 0.5 × 0.5 × 0.5 = 0.125
• More controlled, predictable gradient flow

Why This Matters

When gradients explode (become very large), your parameter updates during training become erratic:

• Update = learning_rate × gradient
• If gradient = 1000, even a small learning rate of 0.01 gives an update of 10
• Your weights jump around wildly, overshooting optimal values
• Training becomes unstable or diverges entirely

With stable gradients from smaller weights, updates are proportional and predictable, leading to smooth convergence.

2. Sensitivity to Input Perturbations

Consider a simple neuron: output = w₁x₁ + w₂x₂ + ... + bias

With large weights (w₁ = 100, w₂ = 150):

• If x₁ changes from 1.0 to 1.01 (tiny noise or measurement error)