17
$\begingroup$

I just finished "An Introduction to Statistical Learning". I wondered whether using cross-validation to find the best tuning parameters for various machine learning techniques is different from data snooping?

We are repeatedly checking which value of the tuning parameter results in a best predictive result in the test set. What if the tuning parameter we arrive at just happens to fit this particular test set by chance, and won't perform well on some future test set?

Please excuse my novice understanding of machine learning, and I'm eager to be educated.

EDIT: Please see @AdamO answer on the definition of "data snooping". I used the term very inaccurately in my question.

Improve this question
$\endgroup$
1
  • $\begingroup$ I am so happy to find this... having a TC tomorrow with some researchers who have published something.... unfortunately suffering from this problem. Great question! $\endgroup$
    – pallevillesen
    Commented May 17, 2017 at 16:19

4 Answers 4

Reset to default
15
$\begingroup$

I wondered whether using cross-validation to find the best tuning parameters for various machine learning techniques is different from data snooping?

Your concern is right spot on, and there is a whole lot of literature on this topic, e.g.

The problem is that hyperparameter tuning with cross validation is a data-driven optimization process, and will still tend to overfit to yor data set (less than tuning by resubstitution error, but still). Trying to use the tuning cross validation results as "independent" performance measure is in a way like eating the pie (= tuning) and keeping (= measure final model performance) it.

This does not mean that you shouldn't use cross-validation for hyperparameter tuning. It just means that you can use it only for one purpose. Either optimize or measure model performance for validation purposes.

The solution is that you need to do an independent validation for measuring the quality of the model obtained with the tuned hyperparameters. This is called nested or double validation. You'll find a number of questions and answers here on these topics

Conceptually, I like to say that training includes all kinds of fancy steps to fit not only the "usual" model parameters but also to fit (auto-tune) the hyperparameters. So data-driven optimization of λ is clearly part of the model training.

As a rule of thumb you can also say that model training is everything that needs to be done before you have a ready-to-use final black-box function that is able to produce predictions for new cases.

PS: I find the testing vs. validation terminology very confusing because in my field "validation" means proving that the final model is fit for purpose, and is therefore what other people call testing rather than validation. I prefer to call the inner test set "tuning test set" and the outer "final validation test set" or the like.

Update:

So if my model (i.e. my tuning parameter in this case) fails the outer validation, what should I do then?

Typically, this is nothing that just happens: there are typical situations that can cause such a failure. And all such situations that I'm aware of are overfitting situations. You need to be aware that while regularization helps to reduce the necessary number of training cases, data-driven optimization needs large amounts of data.

My recommendations:

  • Typically, you (should) already have rough expectations, e.g. what performance should be achievable, what performance you'd consider suspiciously good looking. Or have specs what performance you need to achieve and a baseline performance. From that and the number of availabe training cases (for the splitting scheme you decided for), calculate the expected uncertainty for the inner (tuning) tests. If that uncertainty indicates that you would not be able to get meaningful comparisons, don't do data-driven optimization.

  • You should check how stable both the obtained predictions with the chosen λ and the optimal λ found by the auto-tuning procedure are. If λ isn't reasonably stable with respect to different splits of your data, the optimization didn't work.

  • If you find that either you won't be able to do the data-driven optimization or that it didn't work after all, you may choose the λ by your expert knowledge, e.g. from experience with similar data. Or by the knowledge that if you find out that the optimization failed, you'll need a stronger regularization: the overfitting that leads to the failure works towards too complex models.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ I agree, the terminology train/test/validation is not very intuitive $\endgroup$
    – M. Berk
    Commented Jul 11, 2014 at 16:57
  • $\begingroup$ A nice way to think about it is that the model itself is a hyperparameter too so it makes sense to have a separate set to report the overall performance. $\endgroup$
    – usεr11852
    Commented Oct 20, 2022 at 15:31
3
$\begingroup$

Cross validation is used to assess the validity of a particular finding, usually tangibly related to a prediction model. These kinds of findings can include results from cluster analyses, classification, or prediction models. In any case, the nature of the analysis is part of a pre-specified question of scientific interest such as "which frequencies of mRNA expression most likely originated from breast cancer biopsies versus healthy controls?". Cross validation is a very robust way to assess the validity of a prespecified prediction model that serves a specific purpose. I suspect that $k$-fold cross validation, which involves repeated model fitting, may be the origin of your confusion.

When using CV to identify an "optimal" tuning parameter, as is needed with penalized likelihood methods, there is usually a prespecified criterion that the process is set to meet. This might be minimal MSE, or maximal AUC, or minimal BIC. If you cherry pick a $\lambda$ which gives you results you are after, then you have done something worse than data dredging, I think. So, using a microarray example, if you are interested in which proteins are more expressed in cancer cases versus controls, you might prespecify a GLM LASSO to have $\lambda$ give the best BIC and markers selected in the final model are chosen as candidate proteins for further investigation. This is an example of feature selection.

"Data snooping" or, as I could call it, "Exploratory data analysis" does not deal with a pre-specified question. You kind of enumerate a number of possible, plausibly interesting results and evaluate them individually. You can perform any number of exploratory analyses and, usually, you don't worry about multiple testing. You can assess each exploratory analysis individually using cross-validation, but it does not inherently account for multiple testing when you have more than 1 exploratory analysis. Hypotheses in this setting can be quite wide and far reaching, "which factors are associated with prostate cancer?" (from which coffee drinking, vasectomy usage, etc. etc. were measured in a cohort). Significant results are seen as "hypothesis generating" and provide no confirmatory evidence.

So while both approaches are somewhat "iterative" in nature, they are entirely independent procedures. $k$-fold cross-validation is a tool for assessing uncertainty and validity of a particular set of findings which are part of a "modular" hypothesis. Data dredging, data snooping, or exploratory data analysis is meant to generate hypotheses based on a large set of possibly interesting questions addressed by a comprehensive and large dataset.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thanks for the clarification on "data snooping" -- I think I used it imprecisely in my question. One thing I'm still not clear is whether the "using k-fold CV to cherry-pick $\lambda$ for feature selection" is example of a GOOD or BAD practice? I think it's bad because that particular $\lambda$ may not work again. But on the other hand, that is what the book in the question teaches. $\endgroup$
    – Heisenberg
    Commented Jul 11, 2014 at 16:46
  • 1
    $\begingroup$ @Anh: the tuning of λ by cross validation is not in itself bad. But doing so you "used up" that cross validation for tuning of λ and you need another validation that is independent of the modeling including the process of tuning λ. Not doing this outer validation is BAD. If that particular λ does not work "again" (e.g. on another split of the data), your optimization did not work. In that situation, you'll typically also see a large difference between outer validation results and the observed "best" performance during tuning. $\endgroup$
    – cbeleites
    Commented Jul 11, 2014 at 17:04
  • $\begingroup$ @cbeleites So if my model (i.e. my tuning parameter in this case) fails the outer validation, what should I do then? I can't go back and find another tuning parameter, because essentially that would be turning the outer validation into the tuning test set. So, what to do? $\endgroup$
    – Heisenberg
    Commented Jul 11, 2014 at 17:23
  • $\begingroup$ @Anh I think much depends upon the nature of the scientific question. Going back to the protein binding example, you might observe that a unique set of proteins in the case samples have very uncommon nucleotide frequencies, and thus you would be interested in a small number of possible features in a final model. So you would pick $\lambda$ to heavily penalize the likelihood, or tune it so you identify the top 3 or 4 features in the model. (continued) $\endgroup$
    – AdamO
    Commented Jul 11, 2014 at 19:31
  • $\begingroup$ @anh Then to assess the validity of the $\lambda$ chosen for top 3 features, you would use $k$-fold validation to assess either how many features were selected (if numbers of features vary from none to 20, then there is low evidence of specificity for the 3 features in the training model). The actual tuning parameter, like a likelihood, has a very data-specific meaning. I would never report it, but focus on the sensitivity of results as a consequence of changing it. $\endgroup$
    – AdamO
    Commented Jul 11, 2014 at 19:37
1
$\begingroup$

Actually, during CV you try to find the best parameters on a validation set, which is distinct from the test set. You split your entire data into three sets : training set, validation set, and test set. When you do cross-validation properly, you never look at the test end until the end, so there is no snooping at all. Doing cross-validation on the test set is a serious (yet frequent) methodological error.

Improve this answer
$\endgroup$
3
  • $\begingroup$ If validation set and test set are different then it makes sense to me. However, in the book that I read (by Hastie et al. no less), they argue that using hold-out test is expensive (we're not using lots of data to train), so they recommend k-fold cross validation, in which I don't think there's a separate test set. $\endgroup$
    – Heisenberg
    Commented Jul 11, 2014 at 16:49
  • 1
    $\begingroup$ @Anh: both of the splits can be done by resampling (e.g. repeated cross validation) instead of setting aside only one small data set. $\endgroup$
    – cbeleites
    Commented Jul 11, 2014 at 16:57
  • $\begingroup$ @Anh: In k-fold cross-validation, you split k times the original training set into a smaller training set and a validation set. The original test set is not involved and used only at the end. $\endgroup$
    – Jerorx
    Commented Jul 12, 2014 at 17:02
0
$\begingroup$

If you look for example at page 225 of "An Introduction to Statistical Learning" with Lasso example, you see that they actually do nested cross-validation. I.e. the model selection is done in cv.glmnet on a train set, which is split by the cv.glmnet package into train-test pairs. The model validation is done on validation ("test") set, so it is independent validation.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.