What is the intuitive meaning of generalized weight in a neural net and how does it relate to weight?

Question

For instance, I have implemented a neural net in which the weights at each neuron are non-zero, but the generalized weights for all features of every training example are zero. How is this possible?

Answer 1

I found this article extremely helpful in this regard. It uses R package neuralnet which is the best R implementation I have found so far.

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5009026/

This link is to an open access paper "Neural networks: further insights into error function, generalized weights and others" published by Zhongheng Zhang in August 2016. In the abstract, the authors state "Key concepts of NN including activation function, error function, learning rate and generalized weights are introduced."

Also in the abstract "Generalized weights assist interpretation of NN model with respect to the independent effect of individual input variables. A large variance of generalized weights for a covariate indicates non-linearity of its independent effect. If generalized weights of a covariate are approximately zero, the covariate is considered to have no effect on outcome." (Bold/Italics added for emphasis)

There is much more detail in the paper body. One useful sentence defines generalized weights for neural networks:

"In 2001, Intrator and coworkers developed the concept of generalized weights for the interpretation of NN. The generalized weights are mathematically written as:

where i is the index for each covariate, o(x) is the predicted outcome probability by covariate vector. Log-odds is the link function for logistic regression model. The partial derivative of the log-odds function with respect to covariate of interest is the coefficient for that covariate. However, if there are non-linear terms for the covariate, generalized weights for that covariate vary greatly over the entire covariate pattern. For linear terms as that in conventional logistic regression model, the generalized weights of a covariate are concentrated at one value. "