I have a data set that has 660,000 samples with 72 features and I'm trying to perform feature selection so that I can train a naive bayes classifier. The problem is that since the data set is so big, I can't process the entire file without my computer freezing up. I originally planed on performing feature ranking with information gain by just taking a subsample of the data. The problem is that each time I run my program, I get a different order for the features.

I'm trying to figure out, how large of a percentage of the training data do i need to get an accurate measure for my information gain?

  • $\begingroup$ Check for collinearity in the features. Look at the covariance structure. I.e. the features might be competing with each other. $\endgroup$
    – AnthonyC
    Commented Nov 20, 2015 at 4:10
  • $\begingroup$ Do you mean check the covariance for each pair combination? $\endgroup$ Commented Nov 20, 2015 at 14:00

1 Answer 1


What if you take the average scores(or the average divided by the std) from all runs and then rank the features? Btw I think you should calculate feature importance from the testing set not the training set

  • $\begingroup$ So for example, take a subsample of 5,000 a bunch of times. Let's assume 10,000. Then calculate the information gain for each of these subsamples and divide by the standard deviation? That is similar to bootstrap right? $\endgroup$ Commented Nov 20, 2015 at 14:00
  • $\begingroup$ Yeah. I think its a valid approach for your case. $\endgroup$
    – amanita
    Commented Nov 20, 2015 at 14:05
  • $\begingroup$ Probably you forgot to write it but just in case: First take the average information gain from the 10000 runs for each feature and then divide each feature by its std across runs $\endgroup$
    – amanita
    Commented Nov 20, 2015 at 14:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.