Why is it bad idea to initialize all weight to same value? If all parameters of $\mathbf{W}$ are the same, all $\partial \mathbf{W}$'s rows would also be same. But why is this bad? Can't I proceed from this?
$$
\mathbf{W}=
\begin{pmatrix}
 a & a & a\\
 a & a & a
\end{pmatrix}
,
\frac{\partial L}{\partial \mathbf{W}}=
\begin{pmatrix}
b & c & d \\
b & c & d
\end{pmatrix}
$$
After first update,
$$
\begin{aligned}
\mathbf W &=
\begin{pmatrix}
a -b & a-c & a-d \\
a -b & a-c & a-d
\end{pmatrix} \\
\mathbf X &=
\begin{pmatrix}
x_1 & x_2
\end{pmatrix} \\
\mathbf {XW} &=
\begin{pmatrix}
x_1(a-b) + x_2(a-b) & x_1(a-c) + x_2(a-c) & x_1(a-d) + x_2(a-d)
\end{pmatrix}
\end{aligned}
$$

(Edit)
Belows are example of backpropagation of cross entropy and softmax.


Here $a_1, a_2, a_3$ are input of softmax(=output of last Affine layer) 
$y_1=y_2=y_3$ but $t_1,t_2,t_3$ is all zero except one(one-hot encoding).
So $\frac{\partial L}{\partial a_1}, \frac{\partial L}{\partial a_2}, \frac{\partial L}{\partial a_3}$ is not all same and eventually it causes $\mathbf W$ not to have all same elements after first update.
 A: The optimization algorithms we usually use for training neural networks are deterministic. Gradient descent, the most basic algorithm, that is a base for the more complicated ones, is defined in terms of partial derivatives
$$
\theta_j := \theta_j - \alpha \frac{\partial}{\partial \theta_j} J(\Theta)
$$
Partial derivative tells you how does the change of the optimized function is affected by the $\theta_j$ parameter. If all the parameters are the same, they all have the same impact on the result, so will change by the same quantity. If you change all the parameters by the same value, they will keep being the same. In such a case, each neuron will be doing the same thing, they will be redundant and there would be no point in having multiple neurons. There is no point in wasting your compute repeating exactly the same operations multiple times.
When you initialize the neurons randomly, each of them will hopefully be evolving during the optimization in a different "direction", they will be learning to detect different features from the data. You can think of early layers as of doing automatic feature engineering for you, by transforming the data, that are used by the final layer of the network. If all the learned features are the same, it would be a wasted effort.
You may also be interested in reading the The Lottery Ticket Hypothesis: Training Pruned Neural Networks paper by Jonathan Frankle and Michael Carbin, who explore the hypothesis that the big neural networks are so effective because randomly initializing multiple parameters helps our luck by drawing the lucky "lottery ticket" parameters that work well for the problem.
