I'm currently training a neural network with several hundred input features. I'm pretty sure that only a fraction of them are going to be relevant for predictions, I just don't know which ones those are going to be yet.

Unlike something like image recognition (where if you have one feature you have them all) my features are all individual and there's a small-but-nonzero effort to collecting/calculating each. That — added to the desire for a smaller, potentially more interpretable model — drives my desire to reduce the final model to use a "minimal" number of input features. (I'd also like to reduce/simplify the number of hidden nodes and layers, but this is less of a concern than reducing the number of input features.)

Is there a "clever" way of analyzing a trained neural network to identify the critical features? Or is there a method of (re-)training which will maximize the sparsity of the model?

If it was a simple linear regression, I might reach for L1 regularization/LASSO, but I've run across (unfortunately unsupported) mention that L1 regularization doesn't work well to induce sparsity in neural networks.

Is my only resort to pursue a forward/backward feature selection approach where I drop/add one feature at a time, redo the training, and then compare performance? Or is there some information I can extract from the trained network to aid in this process?

  • $\begingroup$ Someone must indeed have looked into this before. One logical way to compare would be cross - validation or validation set performance with respect to a suitable loss function (i.e. building in the cost of collection of data). I guess one can look at how active/ connected everthing in the input layer is and then manually try to take something out, but whether that's a good thing to do, I do not know. One could also define a new penalty term on how many of the inputs get connected up, no idea whether anyone had done that before. $\endgroup$ – Björn Mar 22 '18 at 6:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.