More specifically, given a typical neural network with a single hidden layer $Z_m$ where $m = 1,...,M$ (see specifications/notation below drawn from p. 392 of Elements of Statistical Learning (Hastie, Tibshirani, Friedman, 2008)), wouldn't the weights for each of the derived features $Z_m$ converge to the same values since each feature $Z_m$ in the hidden layer is derived from the same collection of input observations $X_p$ where $p = 1,...,P$? To my untrained eyes it looks like the hidden layer contain redundant features. What am I missing?

For K-class classification, there are K units at the top, with the kth unit modeling the probability of class k. There are K target measurements $Y_k$, $k = 1,...,K$, each being coded as a 0−1 variable for the kth class.

Derived features $Z_m$ are created from linear combinations of the inputs, and then the target $Y_k$ is modeled as a function of linear combinations of the $Z_m$,

$Z_m = σ(α_{0m} + α^T_mX),$ where $m = 1,...,M,$

$T_k = β_{0k} + β^T_k Z,$ where $k = 1,...,K,$

$f_k(X) = g_k(T),$ where $k = 1,...,K,$

where $Z = (Z_1,Z_2,...,Z_M),$ and $T = (T_1,T_2,...,T_K).$

enter image description here


1 Answer 1


The weights are initialized with different (and typically random) values. Because of this, hidden units will have different activations, and will contribute differently to the output. This breaks the symmetry that you noticed. Because of the asymmetry, weights will converge to different values.

An example

Say we have a 3 layer network. There are $n$ inputs with activations $x = [x_1, ..., x_n]$. There are $n$ hidden units with sigmoidal activations $h = [h_1, ..., h_n] = \tanh(x W + b_h)$. Here, $W$ is a weight matrix, and $b_h$ is a vector of biases. There's a single output unit with linear activation $o = h V + b_o$. We want to predict a target output $t$, and we measure the error with the squared loss $L = (t - o)^2$. We update the weights using stochastic gradient descent with learning rate $\alpha$. This means that, for each training input and target, we calculate the gradient of the loss function w.r.t. each parameter. We then update each parameter by stepping in the direction opposite the gradient.

Consider the parameter $W_{11}$, the weight from the first input unit to the first hidden unit. The update rule is:

$$W_{11} \leftarrow W_{11} - \alpha \frac{\partial L}{\partial W_{11}}$$

The gradient is:

$$ \frac{\partial L}{\partial W_{11}} = \frac{\partial L}{\partial h_1} \frac{\partial h_1}{\partial W_{11}} = 2 V_1 (t - o) x_1 (h_1^2 - 1) $$

In contrast, consider $W_{12}$, the weight from the first input unit to the second hidden unit. The update rule is:

$$W_{12} \leftarrow W_{12} - \alpha \frac{\partial L}{\partial W_{12}}$$

The gradient is:

$$ \frac{\partial L}{\partial W_{12}} = \frac{\partial L}{\partial h_2} \frac{\partial h_2}{\partial W_{12}} = 2 V_2 (t - o) x_1 (h_2^2 - 1) $$

So, we can see that there are some differences. The new value for $W_{11}$ depends on its previous value, $h_1$ (the activation of the first hidden unit), and $V_1$ (the weight from the first hidden unit to the output unit). In constrast, The new value for $W_{12}$ depends on its previous value, $h_2$, and $V_2$.

Because the weights were initialized randomly, the previous values of $W_{11}$ and $W_{12}$ will differ, as will $V_1$ and $V_2$. Because the activations depend on the weights, $h_1$ and $h_2$ will also differ. Therefore, the updated values of $W_{11}$ and $W_{12}$ will differ as well.

  • $\begingroup$ Great answer, thank you. Should the notation for the second gradient be $dL/dW_{12} = dL/dh_2 \times dh_2/dW_{12}$? $\endgroup$
    – RobertF
    Jun 27, 2016 at 16:35
  • $\begingroup$ I see where my logic went wrong. However, if the gradients $dL/dW_{11}$ and $dL/dW_{12}$ depend upon current values of $h_1$ and $h_2$ then the solutions for $W_{11}$ and $W_{12}$ could vary wildly for different starting values of $h_1$ and $h_2$. It seems like either a brute force grid search or manual tweaking would be required to find the optimal model. $\endgroup$
    – RobertF
    Jun 27, 2016 at 16:40
  • 1
    $\begingroup$ Thanks for catching that typo. Regarding search...yes, the loss function is typically nonconvex with many local minima. In practice, gradient-based optimization methods are almost always used, which converge to a local minimum. Stochastic variants that work on individual examples or minibatches can help to escape some local minima. Multiple initializations are also commonly used. Grid search is often used for optimizing hyperparameters (although random search may be better). It would be way too slow for optimizing weights because iterations scale exponentially w/ the number of parameters. $\endgroup$
    – user20160
    Jun 27, 2016 at 21:35
  • 1
    $\begingroup$ This paper may be of interest: Choromanska et al. (2015). The Loss Surfaces of Multilayer Networks). They say that, for large networks, local minima are mostly equivalent in terms of generalization performance (which is not true for smaller networks). Finding the global minimum is hard, and not productive because it leads to overfitting. They also say that saddle points are more of a problem for optimization than local minima. $\endgroup$
    – user20160
    Jun 27, 2016 at 21:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.