A statistical test to measure the importance of features?

Question

I'm currently trying to assess importance of the features for my classifier. The situation is the following: first I train my classifier with all of the features I have and tested on a test set . Then in subsequent experiments I train the classifier with all the features except one feature and then I record by how much the performance decreased. And I repeat the same for all the features. However I have a hyper parameter (window size) in which I repeat the previous evaluation procedure for multiple values for the hyper parameter.

Now the problem that I'm facing is basically that for one value of the hyper parameter the performance decreases when I remove a feature, but for other values of the hyper parameter the performance increases without using the same particular feature. Thus I'm not able to measure how important the feature is.

My question:

1- is there a statistical test which I can use to measure how important the feature is? One trivial thing I can think of is that this: in 7 out of 10 experiments the performance decreased on average by 0.005 with a standard deviation of blah blah blah. Is this a valid way/test (except the blah blah part)?

2- Are there better recommended approaches to accomplish what I'm trying to do?

Answer 1

I assume that you care about prediction (out-of-sample) accuracy.

You are right that your model's hyperparameters and the features you select are inter-related and affect each other. In fact, the features you use could be considered as an additional hyperparameter. Given enough computation power, you might want to tune them together; e.g., for each value of the hyperparameters, perform feature selection.

If this is too expensive, you can try an iterative approach: tune your hyperparameters with all features. Once the hyperparameters are fixed, do feature selection. Then fix the selected features and re-tune the hyperparameters. You can iterate between these two phases until your hyperparameters and features don't change any more.

I should also point out that your approach to feature selection is not the only possible one. If I understand correctly, you are using a step-wise backward heuristic (see, e.g., ESL) but you might want to consider other approaches (see, e.g., this interesting paper).