1
$\begingroup$

I am implementing myself a Neural Network with feedforward and backpropagation with gradient descent to understand better how things work.

After setting up the entire algorithm, I still have a huge doubt. When I initialise the weights, I do it so that each node of the network has its own weight initially assigned. I am doing this for every sample in the dataset, meaning that for each sample I initialise different weights. Is this correct, or should the initial weights be the same across all samples?

To clarify my question with an example: let's say I have a very simple network with just the input and output layer. The input has 2 nodes. The dataset has 2 observations. What I am doing now is that, for each of the observations, I initialise 2 weights and assign them to the input nodes, so that in the end I have 4 different weights. Should I initialise only 2 weights and assign them to their respective nodes in both the observations, so that both nodes will have the same weights value across all my observations?

What I am doing now:

-observation1
   -node1:weight1
   -node2:weight2
-observation2
   -node1:weight3
   -node2:weight4

What I wonder I should be doing instead:

-observation1
   -node1:weight1
   -node2:weight2
-observation2
   -node1:weight1
   -node2:weight2
Improve this question
$\endgroup$
5
  • 2
    $\begingroup$ Don't you just have a set of weights for the network? The network is, after all, an equation with unknown parameters to estimate from the data. The idea of different weights for different observations does not make sense to me for other regression models, and it does not make sense to me for neural networks? (A simple linear regression has a slope and an intercept, and you can calculate them through NN-style gradient descent if you don't feel like using the analytical solution, not one slope and one intercept per observation.) $\endgroup$
    – Dave
    Commented Jul 2 at 14:22
  • $\begingroup$ Thanks, indeed it does not seem to make much sense. After I posted this question I have updated my network to initialise the weights only once and the results makes more sense now, I am super happy about it. I was probably confused by the many dimensions and steps involved in building a neural network from scratch but this indeed was simpler than I thought. $\endgroup$
    – umbe1987
    Commented Jul 2 at 14:26
  • 1
    $\begingroup$ Perhaps you can write a self-answer to "close out" the question. An upside to doing this is that, if you get it wrong, people will comment on or downvote your answer to let you know it is wrong (and then you can delete the answer to recover any lost reputation points). $\endgroup$
    – Dave
    Commented Jul 2 at 14:27
  • $\begingroup$ BTW, I have discovered there was a problem in what I was doing mainly because I could not understand how to make predictions using new data. Now that I have only one set of weights I know how to go about it :) $\endgroup$
    – umbe1987
    Commented Jul 2 at 14:28
  • $\begingroup$ I have self-answered my own question. Will accept it when I can (I need to wait 2 days). Thank you again! $\endgroup$
    – umbe1987
    Commented Jul 2 at 14:33

1 Answer 1

Reset to default
1
$\begingroup$

Thanks to @Dave's comment, I now know that I should initialise the weights only once for all samples (indeed, it does not make much sense to do otherwise).

By the way, just a curiosity: I was confused about this point since, with the way I was doing it, I could not understand how to make predictions on new data (since I ended up having different weights for each training sample). Now that my weights are juts one set for all samples, I can use the trained ones to make predictions on new data easily.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.