7
$\begingroup$

Edit: after skimming this paper6, I narrowed the scope of this question to NLP problems. Relevant excerpt from the abstract (emphasis my own):

We demonstrate that unsupervised preprocessing can, in fact, introduce a substantial bias into cross-validation estimates and potentially hurt model selection. This bias may be either positive or negative and its exact magnitude depends on all the parameters of the problem in an intricate manner.

Motivation

It's obviously wrong to train on test set features with test set labels. But in many ML competitions, it's standard to release test set features and allow participants to train on them. One example is the Real-world Annotated Few-shot Tasks (RAFT) benchmark in NLP.1 Here's an excerpt from the RAFT paper (emphasis my own):

For each task, we release a public training set with 50 examples and a larger unlabeled test set. We encourage unsupervised pre-training on the unlabelled examples and open-domain information retrieval.

In the RAFT competition, you submit predictions by running your model on the same set of unlabeled texts which you may train on. In NLP, a common way to train on unlabeled text is to train a language model which predicts tokens conditional on other tokens.

I understand that releasing test set features is helpful for those hosting the competition, as it allows participants to submit predictions rather than models/code. I also understand that in real-world model development, you may have observed lots of unlabeled text. But I think the critical difference is that in the real world, you don't have access to out-of-sample text.

Question

Is training a model on (in-sample) test set texts, and then evaluating that model on the same test set an optimistic estimator of out-of-sample performance?

A reasonable-sounding hypothesis is that training on (in-sample) test set texts results in correlation between test set predictions and test set labels, which is an optimistic estimator (at least for linear regression, see equation 7.21 in ESL2). But I don't have an argument for how exactly that dependence arises from training on test set texts without test set labels.

The result of my experiment with PCA here has an important implication for ML competitions: if there are few test set observations and features exhibit high rank, then one can artificially reduce error on the test set by fitting a PCA on test set features.

I'm curious to see if a similar type of result can be observed in NLP, where it's standard practice to train language models on unlabeled text before classification tasks.3 I have a feeling that part of the answer lies somewhere in the paper On Causal and Anticausal Learning4 or its child Causal Direction of Data Collection Matters: Implications of Causal and Anticausal Learning for NLP5. These papers establish that semi-supervised learning should only help for data where text causes the target.

References

  1. Alex, Neel, et al. "RAFT: A real-world few-shot text classification benchmark." arXiv preprint arXiv:2109.14076 (2021).

  2. Hastie, Trevor, et al. The elements of statistical learning: data mining, inference, and prediction. Vol. 2. New York: springer, 2009.

  3. Gururangan, Suchin, et al. "Don't stop pretraining: Adapt language models to domains and tasks." arXiv preprint arXiv:2004.10964 (2020).

  4. Schölkopf, Bernhard, et al. "On causal and anticausal learning." arXiv preprint arXiv:1206.6471 (2012).

  5. Jin, Zhijing, et al. "Causal direction of data collection matters: Implications of causal and anticausal learning for NLP." arXiv preprint arXiv:2110.03618 (2021).

  6. Moscovich, Amit, and Saharon Rosset. "On the cross-validation bias due to unsupervised preprocessing." Journal of the Royal Statistical Society Series B: Statistical Methodology 84.4 (2022): 1474-1502.

$\endgroup$
2
  • 1
    $\begingroup$ Digging more re PCA, I found this question. Clearly, there is still confusion on this topic. I checked that the ESL passage is missing a reference, a simulation, or math. $\endgroup$
    – chicxulub
    Commented Apr 7, 2023 at 2:55
  • $\begingroup$ It's s tricky business. I don't know if i have an answer about it somewhere, but you create a dependence between data, that can lead to spurious below chance of above chance performance. $\endgroup$
    – rep_ho
    Commented Apr 7, 2023 at 8:34

2 Answers 2

5
$\begingroup$

TL;DR: seems fine.

I ran experiments with BERT and GPT-2. The experiment code, analysis code, and data (see the Google Drive link in the README) are available here: https://github.com/kddubey/pretrain-on-test/

I recommend reading the (pre-preprint) paper, but I'll leave a short summary of the paper below.


Let's run an experiment on 25 text classification datasets, two LMs whose pretraining data we're pretty sure are not already contaminated, and across a few settings for the number of training and test observations.

For each dataset, the experiment sanity checks that pretraining on unlabeled text (independent of the test set) helps, i.e., that there's an effect to detect. Call this the pretraining boost. Next, the experiment evaluates the bias from pretraining on unlabeled text from the test set instead of unlabeled independent text. Call this the evaluation bias.

Results

The distributions below are those of marginal effects: averaged across 2 LM types, the 25 classification tasks, and their subsamples.

m50

m100

Pretraining on unlabeled text is, on average, clearly beneficial in every setting. Despite this, there isn't evidence of unfairness. The evaluation bias bounces around 0, and is insignificant. At a task level, the results were consistent: pretraining was beneficial in all but 2 of the 25 tasks, and the evaluation bias was positive in 12 of them, negative in 13 of them, and always less than 1% in absolute value.

Meta-analysis

I also wanted to see how (un)stable few-shot benchmarks are for GPT-2 and BERT finetuning. In the experiment above, this variance is exposed by taking up to 100 subsamples from each dataset. What if the experiment took just one subsample? This is what most few-shot benchmarks effectively do.

meta

It turns out that if we sampled 500 unlabeled texts, 100 classification examples, and 500 evaluation examples for each of the 25 datasets, there's a 47% chance that the experiment would've reported a non-negligible positive or negative bias, instead of the close-to-0 bias we found from repeated subsamples. In a really rough sense, including 25 datasets with 1.1k observations each seems like a lot of data. In reality, it'd be no better than flipping a coin.

There's more awareness about variance these days (e.g., https://arxiv.org/abs/2406.10229). And it's probably the case that finetuning GPT-2 or BERT is more unstable than parameter-efficient or prompt-based LLM methods. But I was surprised to see this level of shakiness for my few-shot study.

$\endgroup$
2
$\begingroup$

We need more studies and more thinking such as what @chicxulub wrote above. Speaking in general, unsupervised learning (UL) can lead to false comfort in the structures learned by UL (e.g., principal component loadings may change if computed on a new sample). But the way in which UL leads to supervised prediction makes this primarily unrelated to overfitting. Rephrased, we may not always need to include the UL inside a resampling loop that is used to get an unbiased estimate of model performance (and please don't use classification accuracy as a performance measure). But I need to see studies that actually verify that.

$\endgroup$
1
  • 1
    $\begingroup$ "But I need to see studies that actually verify if" +1. I am on the pessimistic side and would always assume that the UL on test data has an adverse effect on the honesty of my test results - I would also only believe the opposite if the studies evaluated the performance on multiple big dataset and with variois metrics $\endgroup$
    – Ggjj11
    Commented Jan 4 at 18:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.