I have a data set that has 660,000 samples with 72 features and I'm trying to perform feature selection so that I can train a naive bayes classifier. The problem is that since the data set is so big, I can't process the entire file without my computer freezing up. I originally planed on performing feature ranking with information gain by just taking a subsample of the data. The problem is that each time I run my program, I get a different order for the features.

I'm trying to figure out, how large of a percentage of the training data do i need to get an accurate measure for my information gain?

  • $\begingroup$ Check for collinearity in the features. Look at the covariance structure. I.e. the features might be competing with each other. $\endgroup$ – ACE Nov 20 '15 at 4:10
  • $\begingroup$ Do you mean check the covariance for each pair combination? $\endgroup$ – j.jerrod.taylor Nov 20 '15 at 14:00

What if you take the average scores(or the average divided by the std) from all runs and then rank the features? Btw I think you should calculate feature importance from the testing set not the training set

  • $\begingroup$ So for example, take a subsample of 5,000 a bunch of times. Let's assume 10,000. Then calculate the information gain for each of these subsamples and divide by the standard deviation? That is similar to bootstrap right? $\endgroup$ – j.jerrod.taylor Nov 20 '15 at 14:00
  • $\begingroup$ Yeah. I think its a valid approach for your case. $\endgroup$ – amanita kiki Nov 20 '15 at 14:05
  • $\begingroup$ Probably you forgot to write it but just in case: First take the average information gain from the 10000 runs for each feature and then divide each feature by its std across runs $\endgroup$ – amanita kiki Nov 20 '15 at 14:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.