Which features are most relevant to each class in neural for network binary classification? I designed a neural network for binary classification in MATLAB R2015a.


*

*What are differences between two classes? How system detects a sample is from class 1 or 2? For example some if,then functions or some ranges. Something which can help user to have clearer view of the results. For example neural network found that a sample is from class 1. I should say in report that what are differences between in this sample and other samples (class 2) for this results (based on neural network inputs). I know this is a black box but I need more results. I'm checking the trained system using out-of-sample data.

*I want know the effects of inputs on output. Which input (feature) is more impotent (has higher weight) on output of trained neural network.
PS. Suppose that i have 5 inputs (features) that i checked we have higher accuracies with these input's combination. So now i want find class 1 and class 2 characteristics. My neural network has two hidden layers. First one has 5 neurons and second one has 3 neurons. My hidden layers transfer function is 'tansig' and output transfer function is 'softmax' for reporting probabilities for outputs. Now what should i do ?
 A: My first suggestion would be to use Logistic Regression (LR), at least to gain some understanding regarding input feature relationships to output. LR provides a lot more information than neural nets in this regard, since neural nets are much more black boxes.
While neural networks tend to be black boxes, there are  few things things that you can do, depending upon the type of neural network and the problem that you are solving. As an example, consider the image classification. Often the weight vectors for each neuron are examined to look for detection patterns. The results are generally edge detectors, etc.

Similar information is often shown by researchers in a dashboard that is presented at classification time.

In this image, there is also a visual indication of the output by class, indicating likelihood by class (8 being most likely in this image).
You might consider showing examples of inputs that your neural network gets consistently right and wrong when the predicted class is indicated. This gives the user an idea of the types of errors that are likely for the prediction that is made.
A: Regarding the first point, you can either use linear methods that are more explanatory. For example PCA, to visualize and see what feature leads to the increase in variation. Or alternatively a linear model which each coefficient corresponds to each feature. The value of each coefficient is an indicator of how much they are influential.
Regarding the second point, you can use pruning strategy to pin point which layers, inputs are most influential. For example you can use the the prune function. So at every step you remove a node you can calculate the MSE or the loss and see which node in each layer causes the most lost. Alternatively, you can use a linear model like Lasso, as a linear approximation to the Neural Network fit.  
