I'm currently training a neural network with several hundred input features. I'm pretty sure that only a fraction of them are going to be relevant for predictions, I just don't know which ones those are going to be yet.
Unlike something like image recognition (where if you have one feature you have them all) my features are all individual and there's a small-but-nonzero effort to collecting/calculating each. That — added to the desire for a smaller, potentially more interpretable model — drives my desire to reduce the final model to use a "minimal" number of input features. (I'd also like to reduce/simplify the number of hidden nodes and layers, but this is less of a concern than reducing the number of input features.)
Is there a "clever" way of analyzing a trained neural network to identify the critical features? Or is there a method of (re-)training which will maximize the sparsity of the model?
If it was a simple linear regression, I might reach for L1 regularization/LASSO, but I've run across (unfortunately unsupported) mention that L1 regularization doesn't work well to induce sparsity in neural networks.
Is my only resort to pursue a forward/backward feature selection approach where I drop/add one feature at a time, redo the training, and then compare performance? Or is there some information I can extract from the trained network to aid in this process?