Backpropagation with zero weight initialization

Question

Consider the following neural network:

• Input layer is 10 neurons
• Layer 1 is a fully connected layer with 10 output neurons
• Layer 2 is a ReLU activation layer
• Layer 3 is a fully connected layer with a single output
• The label is the sign of the output.

Suppose that we have data which is realizable by this architecture, and for which the labels are balanced (have same number of 0's and 1's), while initializing the SGD with the all zeroes weight vectors. Which of the following statement is correct?

• SGD is guaranteed to converge on convex problems, hence it will always converge to the correct weights.
• The SGD process will not change the weights at all.
• For some data sets, the SGD process will converge to the correct weights.
• This is a non-convex problem, hence the weight will be changed by SGD, but might converge to a local minimum.

So far, I've managed to eliminate the first answer, I'm left with 3 answers which I consider, however, the last answer seems the most logical to me although I'm not an expert on neural networks and therefore I'm asking here this question.

Answer 1

"The SGD process will not change the weights at all."- It will. Why? Check backpropagation formula. There's no reason to not change a weights at all. However, it's better to initialize them with some random values. Why?

Answer's here for this question: Danger of setting all initial weights to zero in Backpropagation

"For some data sets, the SGD process will converge to the correct weights." - It's true. For some it will but initial weight being 0 don't help. It's said that ann with one hidden layer with non-linear activation can model any function so that's why I believe it's true.

I don't get the last statement.

Answer 2

Suppose we are trying to optimize some loss function $\boldsymbol{\epsilon}$ that is parameterized by weights $\boldsymbol{\theta}$. Consider the gradient descent update at time step $(t)$.

$$\boldsymbol{\theta}^{(t+1)} \leftarrow \boldsymbol{\theta}^{(t)} - \alpha \Delta_{\boldsymbol{\theta}}\boldsymbol{\epsilon}$$

where $\alpha$ is the learning rate and $\Delta_{\boldsymbol{\theta}}\boldsymbol{\epsilon}$ is the gradient of the loss function w.r.t. the loss function. Regardless of what value you choose to initalize the weights at, if $\theta_i = \theta_j$ $\forall i,j$, then the gradient signal is identical in all dimensions i.e. $\Delta_{\theta_j}\boldsymbol{\epsilon} = \Delta_{\theta_i}\boldsymbol{\epsilon}$ $\forall i, j$. This means that all the weights will be receiving the same input signal and they will remain equal at all future time steps and therefore no learning can occur.

However, this does not mean that the weights will be unchanged. Indeed, they can change but they will not necessarily lead to a better loss.