16
$\begingroup$

I want to calculate the importance of each input feature using deep model.

But I found only one paper about feature selection using deep learning - deep feature selection. They insert a layer of nodes connected to each feature directly, before the first hidden layer.

I heard that deep belief network (DBN) can be also used for this kind of work. But I think, DBN provides only abstractions (clusters) of features like PCA, so though it can reduce the dimension effectively, I wonder that if it is possible to calculate the importance (weight) of each feature.

Is it possible to calcualte the feature importance with DBN? And are there other known methods for feature selection using deep learning?

Improve this question
$\endgroup$

3 Answers 3

Reset to default
13
$\begingroup$

One approach you can take for almost any prediction model is to first train your model and find its accuracy, then for one input add some noise to it and check the accuracy again. Repeat this for each input and observe how the noise worsens the predictions. If an input is important then the extra uncertainty due to the noise will be detrimental.

Remember set the variance of the noise to be proportional to the variance of the input in question.

Of course noise is random and you don't want one input to appear unimportant due to random effects. If you have few training examples then consider repeatedly calculating the change in accuracy for each training example with a new noise added each time.

In response to the comments:

This analysis can also be done by removing a variable entirely but this has some downsides compared to adding noise.

  • Suppose that one of your inputs is constant, it acts like a bias term so it has some role to play in the prediction but it adds no information. If you removed this input entirely then the prediction would become less accurate because the perceptrons are getting the wrong bias. This makes the input look like it is important for prediction even though it adds no information. Adding noise won't cause this problem. This first point isn't a problem if you have standardized all inputs to have zero mean.

  • If two inputs are correlated then the the information about one input gives information about the other. A model could be trained well if you used only one of the correlated inputs so you want the analysis to find that one input isn't helpful. If you just removed one of the inputs then, like the first point made, the prediction accuracy would decrease a lot which indicates that it is important. However, adding noise won't cause this problem.

Improve this answer
$\endgroup$
9
  • 2
    $\begingroup$ Hugh, I'm familiar with doing that by removing the feature. What are the advantages of replacing the feature with noise? $\endgroup$
    – DaL
    Dec 8, 2016 at 11:22
  • $\begingroup$ @Dan I didn't mean that the feature should be completely replaced with noise, just that some noise should be added. Removing a feature can make unimportant features look important if they have non-zero mean or if they are correlated with other variables. I've edited my answer to explain. $\endgroup$
    – Hugh
    Dec 8, 2016 at 11:52
  • $\begingroup$ Thank you for the comment. But in fact I have a number of feature set(inputs) and many of them are correlated. In this case, maybe the computational time will be close to n! as I need to consider the combinations. So I want to apply deep learning-based model which can consider complex feature combinations. $\endgroup$
    – z991
    Dec 8, 2016 at 12:38
  • $\begingroup$ @z991 in multivariable linear regression the same problem can occur with variables that are not perfectly correlated. Often we introduce each variable one at a time or use all variables and remove them one at a time. There's no way to simply calculate the best combination. If that hasn't been solved for linear regression you won't find a solution for NN's. You could take the same approach and remove variables one at a time and avoid the n! computation. $\endgroup$
    – Hugh
    Dec 8, 2016 at 12:59
  • 1
    $\begingroup$ @Huge Thank you for the comment. I agree with you. But what I really wanted to know was how to calculate the importance of each feature with deep learning, or neural network. As they use several feature extractions(hidden layers), it was difficult for me to analyze the feature importance. It is possible to calculate whole weight of each feature, but it seems to be quite complex and time-consuming. The linked paper used a single linear layer and I think that is a good idea. I wanted to know other better methods for analyzing the feature importance on the network. $\endgroup$
    – z991
    Dec 8, 2016 at 14:48
5
$\begingroup$

Maybe check this paper: https://arxiv.org/pdf/1712.08645.pdf

They use dropout to rank features.

... In this work we use the Dropout concept on the input feature layer and optimize the corresponding feature-wise dropout rate. Since each feature is removed stochastically, our method creates a similar effect to feature bagging (Ho, 1995) and manages to rank correlated features better than other non-bagging methods such as LASSO. We compare our method to Random Forest (RF), LASSO, ElasticNet, Marginal ranking and several techniques to derive importance in DNN such as Deep Feature Selection and various heuristics...

Improve this answer
$\endgroup$
1
$\begingroup$

Have a look at this post: https://medium.com/@a.mirzaei69/how-to-use-deep-learning-for-feature-selection-python-keras-24a68bef1e33

and this paper: https://arxiv.org/pdf/1903.07045.pdf

They present a nice scheme for applying deep models for feature selection.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.