Large weights in neural networks make models highly sensitive because they amplify small changes in inputs, creating instability in several ways:
When you have a large weight w, the output is computed as w·x (simplified). If w = 100 and your input x changes from 1.0 to 1.01, the output swings from 100 to 101 — a tiny 1% input change causes a 1-unit output change. With small weights, this same input variation would barely register.
During backpropagation, gradients are multiplied by weights as they flow backward through layers. Large weights cause gradients to explode exponentially, especially in deep networks. This makes training unstable — the model can overshoot optimal solutions wildly or even diverge entirely.
Large weights push inputs to activation functions (like sigmoid or tanh) into their saturated regions where the function is nearly flat. In these regions, gradients approach zero ("vanishing gradients"), making learning extremely slow or impossible. Meanwhile, small input perturbations near the saturation boundary can cause disproportionate output changes.
Large weights allow the model to fit training data with extreme precision, including random noise. The model becomes hypersensitive to specific training examples rather than learning general patterns. A tiny measurement error or outlier in training data gets magnified into a major feature.
This is why techniques like L2 regularization (weight decay) explicitly penalize large weights, encouraging the model to find smoother, more generalizable solutions that respond proportionally to input variations rather than overreacting to minor fluctuations.