1
$\begingroup$

For instance, I have implemented a neural net in which the weights at each neuron are non-zero, but the generalized weights for all features of every training example are zero. How is this possible?

$\endgroup$

1 Answer 1

1
$\begingroup$

I found this article extremely helpful in this regard. It uses R package neuralnet which is the best R implementation I have found so far.

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5009026/

This link is to an open access paper "Neural networks: further insights into error function, generalized weights and others" published by Zhongheng Zhang in August 2016. In the abstract, the authors state "Key concepts of NN including activation function, error function, learning rate and generalized weights are introduced."

Also in the abstract "Generalized weights assist interpretation of NN model with respect to the independent effect of individual input variables. A large variance of generalized weights for a covariate indicates non-linearity of its independent effect. If generalized weights of a covariate are approximately zero, the covariate is considered to have no effect on outcome." (Bold/Italics added for emphasis)

There is much more detail in the paper body. One useful sentence defines generalized weights for neural networks:

"In 2001, Intrator and coworkers developed the concept of generalized weights for the interpretation of NN. The generalized weights are mathematically written as:

enter image description here

where i is the index for each covariate, o(x) is the predicted outcome probability by covariate vector. Log-odds is the link function for logistic regression model. The partial derivative of the log-odds function with respect to covariate of interest is the coefficient for that covariate. However, if there are non-linear terms for the covariate, generalized weights for that covariate vary greatly over the entire covariate pattern. For linear terms as that in conventional logistic regression model, the generalized weights of a covariate are concentrated at one value. "

$\endgroup$
1
  • $\begingroup$ Welcome to CV. Thank you for your answer. Can you please extend your answer by summarising the content of your link. Link only answers are not welcome as the content of the links might be changed or deleted. $\endgroup$
    – Ferdi
    Commented Nov 6, 2017 at 22:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.