2
$\begingroup$

If all layers' weights are same, hidden layers' weights remain same after updates so NN can't learn. But what happens in neural network if weights of exactly one layer are all same?

$\endgroup$

2 Answers 2

1
$\begingroup$

That one layer acts as one neuron (although the output will get duplicated) and will probably be the bottleneck for information flow. To prevent such cases, some symmetry breaking approaches like dropout are needed together with good random initialisation.

$\endgroup$
0
$\begingroup$

If the weight matrix $W$ is a matrix filled with ones only and multiplied by a weight $w$, then the output is $(W\vec{x})_j=w(\sum_i \vec{x}_i)_j$ (in row j). So this scaled sum is replicated in multiple rows (as often as matrix W has rows).

$\endgroup$
2
  • $\begingroup$ Although this is factually correct, I upvoted the other response because it discusses the implications as well. $\endgroup$ Commented Feb 17 at 21:57
  • 1
    $\begingroup$ You can upvote two answers :) $\endgroup$
    – Ggjj11
    Commented Feb 17 at 22:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.