0
$\begingroup$

hello so this should be a simple question i have a feed-forward neural network i got 10 neurons in a single hidden layer i obtained it's weights now there are some weights positive at values up to 1.1 and values at negative of down to -1.3 and values super small in both negative and positive side of like 0.00000something so i understand basic concept bigger weights more influence. the question is what does the negative represent so like is -1.3 higher influence than a positive +1.1? and i should ignore the signs completely ? or are they Negative < small negative < small positive < positive kind of system?

$\endgroup$
1
0
$\begingroup$

It is indeed the magnitude of the weights that are the indicator of importance, a negative weight is just an inhibitory influence on the neuron that it feeds, tending to prevent it from firing. The signs of the weights is not always that easy to interpret. If you have a multi-layer perceptron with one hidden layer, IIRC if you invert all of the input layer weights and all of the output layer weights (assuming a tanh activation function in the hidden layer), the function of the network will remain exactly the same.

Note also that large magnitude weights are often involved in over-fitting, so may not reflect real importance, just exploiting some noise in the training set. Interpretation of weights of neural nets is generally rather tricky (IMHO).

$\endgroup$
2
  • $\begingroup$ thank you a lot just a small follow up question how big of a magnitude are we talking that i can start to suspect over-fitting? or is it not a kind of a constant thing $\endgroup$ Jul 8 at 8:29
  • $\begingroup$ I think it will depend a lot on network architecture, if you have a lot of hidden neurons, they can distribute the same behaviour across multiple units, while keeping the output layer weights small. IIRC the size of the output layer weights tends to be more important (see ieeexplore.ieee.org/abstract/document/661502 but I haven't read it for a long time and it is quite theoretical). I tend to use regularisation (weight decay) when training neural networks. $\endgroup$ Jul 8 at 12:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.