Weight initialisation for neural networks - should they be different for each observations or the same?

I am implementing myself a Neural Network with feedforward and backpropagation with gradient descent to understand better how things work.

After setting up the entire algorithm, I still have a huge doubt. When I initialise the weights, I do it so that each node of the network has its own weight initially assigned. I am doing this for every sample in the dataset, meaning that for each sample I initialise different weights. Is this correct, or should the initial weights be the same across all samples?

To clarify my question with an example: let's say I have a very simple network with just the input and output layer. The input has 2 nodes. The dataset has 2 observations. What I am doing now is that, for each of the observations, I initialise 2 weights and assign them to the input nodes, so that in the end I have 4 different weights. Should I initialise only 2 weights and assign them to their respective nodes in both the observations, so that both nodes will have the same weights value across all my observations?

What I am doing now:

-observation1
-node1:weight1
-node2:weight2
-observation2
-node1:weight3
-node2:weight4


What I wonder I should be doing instead:

-observation1
-node1:weight1
-node2:weight2
-observation2
-node1:weight1
-node2:weight2

• Don't you just have a set of weights for the network? The network is, after all, an equation with unknown parameters to estimate from the data. The idea of different weights for different observations does not make sense to me for other regression models, and it does not make sense to me for neural networks? (A simple linear regression has a slope and an intercept, and you can calculate them through NN-style gradient descent if you don't feel like using the analytical solution, not one slope and one intercept per observation.)
– Dave
Commented Jul 2 at 14:22
• Thanks, indeed it does not seem to make much sense. After I posted this question I have updated my network to initialise the weights only once and the results makes more sense now, I am super happy about it. I was probably confused by the many dimensions and steps involved in building a neural network from scratch but this indeed was simpler than I thought. Commented Jul 2 at 14:26
• Perhaps you can write a self-answer to "close out" the question. An upside to doing this is that, if you get it wrong, people will comment on or downvote your answer to let you know it is wrong (and then you can delete the answer to recover any lost reputation points).
– Dave
Commented Jul 2 at 14:27
• BTW, I have discovered there was a problem in what I was doing mainly because I could not understand how to make predictions using new data. Now that I have only one set of weights I know how to go about it :) Commented Jul 2 at 14:28
• I have self-answered my own question. Will accept it when I can (I need to wait 2 days). Thank you again! Commented Jul 2 at 14:33