It is apparently a common practice to use ROC as a feature selection method in my new job. They test the variables one by one against the response and anything with ROC<=54 is tossed aside.

Would you say that this is a good practice? I'm quit skeptical as I would have rather used ensemble learning in order to incorporate feature selection. Or used a regularization method like lasso/elastic net.

  • $\begingroup$ Not sure i follow. They are fitting one variable models, computing the auc of this model, then discarding? $\endgroup$ – Matthew Drury Feb 14 '17 at 1:44
  • $\begingroup$ @MatthewDrury that's pretty much how it goes. Any AUC over 54 is okay to keep for further subsequent analyses. $\endgroup$ – Herzriesig Feb 14 '17 at 16:09
  • $\begingroup$ Isn't that pretty time consuming? I mean, where I work, if we tried to do that it would take months for us to come out with a useful model. How large is the training data? $\endgroup$ – Jon Feb 14 '17 at 16:12
  • $\begingroup$ @jon It is extremely time consuming! The training data I'm working with consists of some hundred thousand instances with less than 100 predictors. However, the usual dataset is much smaller. More so, I don't know if it is a legit feature selection procedure. I have never heard of anyone doing this before at least $\endgroup$ – Herzriesig Feb 14 '17 at 16:22
  • 1
    $\begingroup$ @Nesvold I'm sorry to hear that. Every SAS program is a small tragedy. $\endgroup$ – Matthew Drury Feb 14 '17 at 16:47

Univariate feature selection is generally a poor method.

This question is deftly answered by silverfish in the context of correlation, but all his arguments apply to your case as well. In short, there is no reason to believe that univariately checking how each individual variable $x$ is related to your response $y$ reveals anything about the multivariate nature of the relationship between $X$ and $y$. It's quite possible that you end up screening out many of your good predictors.

As you point out, LASSO, ridge, or glmnet are much preferred methods for feature selection in a multiple regression model, as they:

  • Take a fully multivariate view of your predictor / reponse relationship.
  • Avoid making high variance, binary decisions like "this variable is completely in, this variable is completely out".
  • Lend themselves naturally to cross-validation and other model validation techniques.

You should carefully and respectfully start pointing your team towards a more modern and disciplined approach.

(*) You also don't mention if your team is testing for non-linear relationships when fitting these univariate models. At the very least, these univariate models should be based on some basis expansion of the feature, like cubic splines. Clearly if they are only testing for univariate linear relationships, there there are some issues there as well.

| cite | improve this answer | |
  • $\begingroup$ Thanks man! My thoughts exactly! Now if I only could convince these conservative SAS analyst about this and that we need to scrap SAS asap, then things would be a-okay $\endgroup$ – Herzriesig Feb 14 '17 at 17:10
  • $\begingroup$ There are no tests with splines I'm afraid, but that's a good suggestion. $\endgroup$ – Herzriesig Feb 15 '17 at 6:44
  • $\begingroup$ "Avoid making high variance, binary decisions like "this variable is completely in, this variable is completely out"." - True for $\ell_2$-regularization, but $\ell_1$-regularization induces sparsity doing exactly that. $\endgroup$ – Firebug Feb 16 '17 at 16:00
  • $\begingroup$ @Firebug Right, and it's known to be high variance, hence the use of both in ridge and LASSO in glment. On the other hand, at least you have a continuous parameter to tune how much you are doing that, and shrinks the parameters gradually before completely removing them, so it's better than going completely on your own. $\endgroup$ – Matthew Drury Feb 16 '17 at 16:09
  • 1
    $\begingroup$ @Firebug Thanks for the tip on terminology, I can't keep it straight. $\endgroup$ – Matthew Drury Feb 16 '17 at 16:10

Echoing Matthew's response from another light: many markers have a concept of stratified predictive accuracy. In this sense they provide extremely good predictive accuracy in a subgroup or in-tandem with another marker. Two examples from the health sciences:

Suppose for instance two types of breast cancer grow in women who are premenopausal and post-menopausal. The majority of women who are diagnosed with cancer are post-menopausal, yet the cancers diagnosed in pre-menopausal women are extremely aggressive, difficult to treat, and their genetic markers are unknown. If you naively presume that all cancers have the same genotype and genetic markers, you will show a low predictive accuracy for a genetic marker that yields a 100% Area-Under-The ROC in premenopausal women.

Another example is how a dyad of conditions might be necessary for disease. For instance, in nephrology, people are typically at risk and subsequently are diagnosed with Stage 3 Chronic Kidney Disease once they develop both hypertension and diabetes. CKD is known to progress to ESRD, which required chronic renal replacement therapy, and is generally very bad. Interventions to halt progress of CKD are poorly known, so earlier diagnostics are needed, but the manifestations of the disease are coincident with many other conditions, it is only a specific spectrum of conditions or a combination of markers that may inform clinicians that the patient has CKD.

Just the same, a stepwise approach to evaluating a sequence of markers does not hold the promise of identifying an optimal ROC in any scientific sense, albeit perhaps in a statistical sense it would. In my experience of evaluating markers, the best use for ROC is evaluating a pre-specified hypothesis about a specific marker rather than a huge list thereof. The CIs for ROCs and their AUCs tend to be quite wide since they deal with empirical functions, and after correction for multiple testing, the risk of Type II error is just too high to justify evaluating 100 markers or more.

The exception to this might be the understanding that the nature of the analysis is a hypothesis generating study. That means, however, that none of the previously collected data may serve to confirm this hypothesis. A separate study must be done. Many clinicians are disheartened at how irreproducible results can be from such fishing expeditions.

| cite | improve this answer | |
  • $\begingroup$ Interesting read! :) $\endgroup$ – Herzriesig Feb 14 '17 at 19:28

Since you are using SAS, I thought I'd share this. I'm not sure what model you are using, but if you are using logistic regression this may be a useful resource.

Sample 54866: Logistic model selection using area under curve (AUC) or R-square selection criteria

Under "Details", it reads:

In addition to the AIC and BIC criteria available in PROC HPLOGISTIC, the SELECT macro can also choose models using the area under the ROC curve (CHOOSE=AUC), the R-square statistic (CHOOSE=RSQUARE), or the max-rescaled R-square statistic (CHOOSE=RSQUARE_RESCALED).

Under "Limitations", it reads

The best subsets selection method (SELECTION=SCORE) available in PROC LOGISTIC is not available in the SELECT macro. The area under the ROC curve criterion (CHOOSE=AUC) is not available with nominal, multinomial (LINK=GLOGIT) models.

| cite | improve this answer | |
  • $\begingroup$ @Nesvold is logistic regression the model you are using? $\endgroup$ – Jon Feb 14 '17 at 23:15
  • 1
    $\begingroup$ These procs implement stepwise selection methods, which are generally ill advised. Should we really be recommending them to the OP? $\endgroup$ – Matthew Drury Feb 14 '17 at 23:50
  • $\begingroup$ That's a good question. However, the tool wouldn't be implemented by SAS (or other software) if it did not have its merits/purpose. $\endgroup$ – Jon Feb 15 '17 at 0:03
  • $\begingroup$ Also, by OP's comments on their model selection methodology, I would think stepwise might provide a more efficient method of model selection. It seems they're already using naive methods for model selection, so why not let SAS do brunt of the work load. $\endgroup$ – Jon Feb 15 '17 at 0:05
  • $\begingroup$ @Jon I don't think that's necessarily true, as it ignores the influence of time. A lot of the basic SAS procs were written a long time ago, and their methodologies have been supplanted by more recent methods. There are plenty of bad methods out there, and code written that implement them. $\endgroup$ – Matthew Drury Feb 15 '17 at 0:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.