why small weights are preferred in neural networks

Question

I'm viewing the CS231n lectures and trying to understand some of the regularization concepts. I think I understand the rationale behind "spread out" weights that are preferred by L2 regularization, e.g. regularization penalty is higher for weights [1, 0] than for [0.5, 0.5]. Intuitively, the latter set of weights would use more of your input features. But why does the absolute value of weights matter, as opposed to only their relative values? I've heard that using L1 regularization helps with "feature selection" because in preferring small weights, some eventually shrink very close to zero. But wouldn't the same effect be observed if some weights were just relatively much smaller?

Answer 1

The magnitudes matter.

Recall that the operation a neuron performs is y=A(wx+b), where A is the activation function, w is weight, b is bias and x is, of course, the input. This means the multiplication by the weight kernel occurs before computing the activation.

Now, if you were using only linear activations, then it indeed wouldn't matter - the derivative of a linear function is a scalar constant, so by changing the weights you're just rescaling the outputs.

For NNs though, what you want out of your deep layers is non-linearity. That's where the magnitudes of weights start to matter.

In most activation functions, very small weights get more or less ignored or are treated as 'evidence against activation' so to speak.

On top of that some activations, like sigmoid and tanh, get saturated. A large enough weight will effectively fix the output of the neuron to either the maximum or the minimum value of the activation function, so the higher the relative weights, the less room is left for subtlety in deciding whether to pass on the activation or not based on the inputs.