i see the expression "principled feature selection" in titles of various Machine Learning papers and generally in the literature but nowhere do authors really define what they mean. "principled" as opposed to? what's the difference, for example, from "regular" feature selection? and which ML/statistical models apply which?
-
2$\begingroup$ Could you link to some examples? The context seems to be important here, since the authors are likely contrasting two approaches. $\endgroup$– Sycorax ♦Commented May 8, 2016 at 20:46
-
$\begingroup$ @C11H17N2O2SNa here are two examples: dsl-lab.org/ml_tutorial_old/Publications/aistats2003.pdf and arxiv.org/pdf/1312.5869v2.pdf $\endgroup$– stas gCommented May 8, 2016 at 21:11
-
1$\begingroup$ @HalilPazarlama definitely not $\endgroup$– stas gCommented May 8, 2016 at 21:12
1 Answer
I would agree that a precise definition is hard to come by. My understanding (which others, e.g., seminar participants also seem to agree with) is the following: "principled" refers to the fact that you let yourself be disciplined by an algorithm/a procedure (to for example choose tuning parameters) to select your predictors rather than handpick them so as to, for example, produce an impressively low error on your training data set.
Similarly, if you were (which is not so often the case in ML) interested in confirming some theory, an unprincipled way of choosing predictors would be to try many models until one comes out where your coefficients of interest have the desired sign and statistical significance.
-
1$\begingroup$ thank you, that's a good starting point to do some more digging about $\endgroup$– stas gCommented May 9, 2016 at 10:12
-
5$\begingroup$ That may be one definition of principled but it may cause misperceptions. Often the most principled approach is based on having experts select the predictors. We also need to remember that feature selection is unnecessary in many cases, and is often in competition with optimal predictive discrimination. $\endgroup$ Commented May 9, 2016 at 12:34