Weight of neural network and feature importance I have constructed a neural network with training dataset.
The classification performance was quite good, but I wonder wehter there is a way to find that which inputs were more relevant in classifying test samples.
As the neural network is a weighted graph, I think calculating the whole weights of each input mean feauture importance, but is it proper?
Suppose it contains 2 hidden layers(h1, h2) and both hidden layers contain 2 nodes(n11, n12, n21, n22). The number of input variables is two(X, Y).
For example, for two inputs X and Y, suppose the weight of each edge is like below.
weight of X to n11 = 0.1 
weight of X to n12 = 0.2
weight of Y to n11 = 0.2
weight of Y to n12 = 0.3
weight of n11 to n21 = 0.4
weight of n12 to n21 = 0.6

I think the weight can be simply calculated.
weight of X = 0.1*(0.4+0.6) + 0.2*(0.4+0.6) = 0.3
weight of Y = 0.2*(0.4+0.6) + 0.3*(0.4+0.6) = 0.5

In this case, can I say that input Y is better feature than input X for classification with this neural network?
 A: I am no expert, but I think you must calculate the gradient of the loss function wrt the inputs.  This would tell you the relative importance of each input for classifying a specific training example. To get a measure of importance of an input for the whole training set you could take the average of these gradients across all examples in the training set.  Or it might be better to take the MAX of the gradients (a low value would mean an input never had much effect).
A: You haven't mentioned the weights of nodes n11 and n12 on n22 respectively.
the dimension of first hidden layer is (2,2) and suppose the dimension of 2nd hidden layer is (2,1) assuming we are working on binary classification, then you can just do the matrix multiplication of weights and that will give you the cumulative weight of each feature on the output. I don't know whether that's the correct way to find the important features but atleast I am getting results which are similar to the results of Lasso and RandomForest feature extraction.
