I'm training a (regression) learner on a $p \gg n$ problem, including bagging and filter feature selection (information gain).

I'm in doubt though regarding the order of the procedures:

  • Apply the filter first, then bagging randomly select features from these only.

  • Apply the filter after randomly selecting features for bagging.

I can see the filter reduces the randomization of bagging if applied first. Still, I'm more inclined towards the first approach. Is there any argument against/for either?

Edit: I can see the second option should take into consideration computational resource constraints as well, but let's focus on the statistical implications of such choice.

  • $\begingroup$ Bagging often refers to repeatedly fitting a model on bootstrapped data points, then averaging the results. But, some people call randomly selecting a subset of features 'attribute bagging'. It sounds like you're referring to the second case, is that right? $\endgroup$ – user20160 Jul 4 '16 at 9:14
  • $\begingroup$ @user20160 Yes, I didn't mention it because it can be a general bagging heuristic, i.e. random subsets of both samples and features are allowed (random forests are like that), so the question is more general. But only subsets of the features have any relation to this question. $\endgroup$ – Firebug Jul 4 '16 at 11:31
  • $\begingroup$ Do you know if all features should contain some useful information, or could there be useless/random features too? And how strong is the inter-feature correlation? I think those could influence the order. BTW: interesting question IMHO. $\endgroup$ – geekoverdose Jul 4 '16 at 12:44
  • $\begingroup$ @geekoverdose it's a $p \gg n$ problem with many useless features, but no random features. The inter-feature correlation follows a normal distribution (mean 0.03 and standard deviation 0.2). The quantiles of the correlations are q(0%, 25%, 50%, 75%, 100%) = (-0.82 -0.11 0.03 0.17 0.98). $\endgroup$ – Firebug Jul 4 '16 at 13:16
  • $\begingroup$ Just to nit-pick, is it reasonable to describe a variable with finite support as following a normal distribution? Something like beta may be a better choice. $\endgroup$ – Andris Birkmanis Oct 26 '18 at 16:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.