Question: Is it expected that when computing the weight-based importance of inputs to outputs in multilayer multioutput perceptrons that the weight importance would be identical in magnitude but with different signs?
I'm attempting to extract information from a supervised neural network using some of the approaches detailed in Olden's "Illuminating the Black Box". Specifically I'm experimenting with the analysis of networks weights he describes as Garson's algorithm.
A bit about my architecture: I'm using synaptic js for my basic architecture. The networks I'm experimenting on are successful multilayer perceptrons with 1 or 2 hidden layers which have been trained on a variety of regression problems. The implementation of Garson's algorithm is something I wrote, which is why I have this question. Note that my approach differs slightly from the one in that paper because I retained the sign during all the multiplications.
The analysis of weights seems accurate and useful for networks with a single output neuron. For networks with no hidden layer, it also seems accurate, so I believe the implementation is correct. However, if there are multiple inputs and at least one hidden layer, all the magnitudes of importance are the same but the signs differ. See image below. The first row is input names, the second is output neuron 1, the third is output neuron 2. This network has one hidden layer of 10 neurons. Importance is rounded for readability, but they are identical out as many decimal places as desired.
Here is a similar network with no hidden layer trained on the same dataset. Note that the relative importances differ dramatically between the output neurons, but the overall importance remains similar to the multilayer network.
Is this the expected outcome? I've checked the math and it seems correct. The last hidden layer has different numbers for each input for each neuron, but the weights in the final layer make all the numbers the same. I'm looking for confirmation either from a theoretical or experience standpoint that this behavior is expected, or confirmation that I messed up the math.
Vaguely similar: Extracting weight importance from One-Layer feed-forward network, but different as this takes the approach described in the answer and specifically focuses on multiple outputs.
