1
$\begingroup$

If you run k-fold cross-validation, and you do not just take the mean of the accuracies but instead, you take the dataset split with the best validation accuracy to use this split as a static split of training and testing set, what kind of "bias in research" would this resemble in classical hypothesis testing?

The question is about the right terminology, a short explanation or best an example. Some possible statistical biases are listed in Bias in research, quoted in in the following example (some other biases might be for example in Identifying and Avoiding Bias in Research as well).

Example answer

Terminology:

  • Sampling error in hypothesis testing
  • Bias in data analysis
  • "Fishing for P"

Short explanation:

performing multiple testing (“fishing for P”) by pair-wise comparisons (4), testing multiple endpoints and performing secondary or subgroup analyses, which were not part of the original plan in order “to find” statistically significant difference regardless to hypothesis.

Example:

For example, if the study aim is to show that one biomarker is associated with another in a group of patients, and this association does not prove significant in a total cohort, researchers may start “torturing the data” by trying to divide their data into various subgroups until this association becomes statistically significant. If this sub-classification of a study population was not part of the original research hypothesis, such behavior is considered data manipulation and is neither acceptable nor ethical. Such studies quite often provide meaningless conclusions such as:

CRP was statistically significant in a subgroup of women under 37 years with cholesterol concentration > 6.2 mmol/L;

lactate concentration was negatively associated with albumin concentration in a subgroup of male patients with a body mass index in

the lowest quartile and total leukocyte count below 4.00 × 109/L.

Besides being biased, invalid and illogical, those conclusions are also useless, since they cannot be generalized to the entire population.

$\endgroup$
3
  • 1
    $\begingroup$ To use the lingo of the second paper you included, this is flawed study design. $\endgroup$ Jun 4, 2021 at 15:12
  • $\begingroup$ @AryaMcCarthy I see, it is in the Fig. 1 there. If that is true, the tag "selection-bias" is wrong. I will take it out. But "Flawed study design" is not very detailed. It can be the most detailed term here, but perhaps there is more? $\endgroup$ Jun 4, 2021 at 15:21
  • 1
    $\begingroup$ You can think of it as a sampling bias, if you want. I’ve updated my answer to reflect this. The taxonomy isn’t sacrosanct; categories can blur with each other. $\endgroup$ Jun 4, 2021 at 15:29

1 Answer 1

2
$\begingroup$

To use the lingo of the second paper you included, this is flawed study design. You could see this flawed design as inducing a sampling bias.

By taking the maximum over all folds, you’re searching for an optimistic score on the test set. You’ll get that from a beneficially constructed test set.

This breaks the iid assumption about how the training and test set items are created. The highest score probably comes from a greater similarity between the training and test sets (among the $k$ train/test splits) or the absence of harder cases from the test set.

$\endgroup$
3
  • $\begingroup$ I see that I made a mistake in the question. I really meant just to make the "best validation set" static - after trying k-fold - and not to use that validation set as the testing set then. But since I have asked for the best split into training and testing set, I accept this answer now, which is not bad to ask for either. Anyway, the testing set was not the reason to ask, just an additional question. When using the validation set, you will always just estimate the generalisation error. I adjusted the header accordingly to ask for the "testing set" split so that your answer fits. $\endgroup$ Jun 4, 2021 at 16:47
  • $\begingroup$ It’s still flawed design, then. You perform model selection based on a bad rule. $\endgroup$ Jun 4, 2021 at 16:55
  • $\begingroup$ I will not ask the other question anymore, as this here is technically already a large part of what the other question would be about, as you say it. $\endgroup$ Jun 4, 2021 at 17:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.