I would like to find a good pair of predictors out of about 400 available pairs. To do this I am using LOO cross validation. Since there are so many pairs available, don't I run into the issue that the best pair could be a spurious finding? I'm thinking this is much like multiple testing.

To be more concrete, for a given pair of predictors I perform LOO cross validation to estimate the loss function using those predictors. That is, for each of the n folds I leave out a sample, build the model using the given predictor pair, and then compute its loss function on the held out sample. These n loss values are then averaged to determine the merit of the predictor pair. I select the pair with the lowest average loss.

Much of what I have read about CV in optimizing models is to optimize over a few parameters (like lambda in Lasso) rather than many (e.g. whether to include a feature). So is there a rule of thumb for how many such tests one can perform with CV?

  • $\begingroup$ What do you mean by "pair of predictors"? It isn't a problem to use the LASSO with an arbitrarily large number of predictors to select only 2, but those 2 wouldn't necessarily be part of some pre-specified pair. $\endgroup$ Jul 17, 2015 at 16:19
  • $\begingroup$ Let's say I have 30 predictors and want a classifier that uses two of them in a glm or other model. Then I have (30*29/2)=435 possible pairs, any of which would be acceptable. I'm wondering if that number, 435, is such a large number of different cross validations that you are bound to get a spuriously low loss that doesn't actually validate.Thanks for the LASSO info; I will try adjusting lambda to get 2 predictors... $\endgroup$
    – OncoStats
    Jul 17, 2015 at 19:45
  • $\begingroup$ Why are you limited to a single pair of predictors? $\endgroup$
    – EdM
    Jul 17, 2015 at 20:11
  • $\begingroup$ For the moment just because it is easy to visualize on a scatter plot. Longer term, fewer predictors is better since we would need to purchase antibodies to make the measurements. We'd like to keep it cheap. :) $\endgroup$
    – OncoStats
    Jul 17, 2015 at 20:30

2 Answers 2


don't I run into the issue that the best pair could be a spurious finding? I'm thinking this is much like multiple testing.

Yes, you are right: this is a multiple testing situation.

Here are some things you can do in order to still draw valid conclusions.

  • Before starting on the comparison, do a rough calculation whether your sample size allows you to do comparisons, e.g. by calculating binomial confidence intervals for some typical outcomes of the testing. This is particularly critical in case of classification if you measure and compare performance by sensitivity, specificity, overall accuracy or in fact any proportion of tested cases.
    If this indicates that you cannot do even single comparisons, you should refrain from data-driven optimization and go directly for a suitable regularization, e.g. the LASSO or PLS (see below).

  • Compare using a so-called proper scoring rule (they are "better behaved" than those proportions and have among other beneficial statistical properties like reacting continuously and sensitively to slight improvements also lower variance.

  • All conclusions you draw about the predictive ability of the chosen pair of predictors should be drawn from independent test data. So either set aside a test set to validate the final two-predictor-model (if you have enough cases), or use a nested (aka. double) cross valiation / out of bootstrap: the inner loop to optimize the predictor pair, the outer loop to validate the final model.

  • Compare this outer validation performance to the "inner" optimization performance. If there's a large discrepancy (particularly if the optimization loop performance was (nearly) perfect), the optimization probably did not work, and you ended up more or less accidentally with a combination that just looks well for LOO and your data set.

  • check whether there were many predictor combinations that all yielded practially the same optimal performance

  • As the outer testing was nevertheless independent, even though you cannot claim that those 2 predictors are the best among the studied, you can still claim that you observed that performace on unknown test individuals for this predictor combination.

  • Even though for the LASSO typically λ is optimized using an inner optimization CV/bootstrap for optimal predictive performance, this is not necessary in your case: you just need a λ so that 2 coefficients stay non-zero. Finding this λ does not require any performance testing, so you can do this without the need for a nested validation design.

  • If the restriction is only because of visualization, you could also try PLS instead of the LASSO. With PLS you can select beforehand that you want to have the first 2 latent variables only, i.e. the two linear combinations that are (according to the PLS criterion) most useful for prediction. In contrast to the LASSO, PLS tends to include most predictors with differing weights into linear combinations and works well in situations with highly correlated predictors, so you'd need to think whether this makes sense for your data qnd application.
    (As few as possible antibodies sounds more like LASSO, though)

  • $\begingroup$ Thanks @cbeleites. I had to spend a little time on Wikipedia to understand them, but these comments are helpful. I think I see your point on the LASSO. Let me be sure I understand, though. Basically you are saying to use LASSO to do the variable selection (it is essentially a one-parameter "fit" so doesn't suffer from the multiple comparisons issue). Then once those are chosen I can find the optimal classifier based on those two variables (e.g. using glm). By nesting this process within a CV framework I can also estimate my performance. Sound like an OK approach? BTW, is there a book to read? $\endgroup$
    – OncoStats
    Jul 20, 2015 at 20:48

This is the wrong way to do cross-validation.

Cross-validation should be used to select the best process for selecting important features. The feature-selection process should only have access to the training set of each fold. You then select the best process based on the performance on the validation folds.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.