Is pretraining on test set texts (without labels) ok?

Question

Edit: after skimming this paper⁶, I narrowed the scope of this question to NLP problems. Relevant excerpt from the abstract (emphasis my own):

We demonstrate that unsupervised preprocessing can, in fact, introduce a substantial bias into cross-validation estimates and potentially hurt model selection. This bias may be either positive or negative and its exact magnitude depends on all the parameters of the problem in an intricate manner.

Motivation

It's obviously wrong to train on test set features with test set labels. But in many ML competitions, it's standard to release test set features and allow participants to train on them. One example is the Real-world Annotated Few-shot Tasks (RAFT) benchmark in NLP.¹ Here's an excerpt from the RAFT paper (emphasis my own):

For each task, we release a public training set with 50 examples and a larger unlabeled test set. We encourage unsupervised pre-training on the unlabelled examples and open-domain information retrieval.

In the RAFT competition, you submit predictions by running your model on the same set of unlabeled texts which you may train on. In NLP, a common way to train on unlabeled text is to train a language model which predicts tokens conditional on other tokens.

I understand that releasing test set features is helpful for those hosting the competition, as it allows participants to submit predictions rather than models/code. I also understand that in real-world model development, you may have observed lots of unlabeled text. But I think the critical difference is that in the real world, you don't have access to out-of-sample text.

Question

Is training a model on (in-sample) test set texts, and then evaluating that model on the same test set an optimistic estimator of out-of-sample performance?

A reasonable-sounding hypothesis is that training on (in-sample) test set texts results in correlation between test set predictions and test set labels, which is an optimistic estimator (at least for linear regression, see equation 7.21 in ESL²). But I don't have an argument for how exactly that dependence arises from training on test set texts without test set labels.

The result of my experiment with PCA here has an important implication for ML competitions: if there are few test set observations and features exhibit high rank, then one can artificially reduce error on the test set by fitting a PCA on test set features.

I'm curious to see if a similar type of result can be observed in NLP, where it's standard practice to train language models on unlabeled text before classification tasks.³ I have a feeling that part of the answer lies somewhere in the paper On Causal and Anticausal Learning⁴ or its child Causal Direction of Data Collection Matters: Implications of Causal and Anticausal Learning for NLP⁵. These papers establish that semi-supervised learning should only help for data where text causes the target.

References

1. Alex, Neel, et al. "RAFT: A real-world few-shot text classification benchmark." arXiv preprint arXiv:2109.14076 (2021).

2. Hastie, Trevor, et al. The elements of statistical learning: data mining, inference, and prediction. Vol. 2. New York: springer, 2009.

3. Gururangan, Suchin, et al. "Don't stop pretraining: Adapt language models to domains and tasks." arXiv preprint arXiv:2004.10964 (2020).

4. Schölkopf, Bernhard, et al. "On causal and anticausal learning." arXiv preprint arXiv:1206.6471 (2012).

5. Jin, Zhijing, et al. "Causal direction of data collection matters: Implications of causal and anticausal learning for NLP." arXiv preprint arXiv:2110.03618 (2021).

6. Moscovich, Amit, and Saharon Rosset. "On the cross-validation bias due to unsupervised preprocessing." Journal of the Royal Statistical Society Series B: Statistical Methodology 84.4 (2022): 1474-1502.

Answer 1

TL;DR: seems fine.

I wrote a paper to answer this question. It appeared in the GenBench Workshop at EMNLP 2024. Here's a short-ish summary.

---

First, let's run an experiment on BERT and GPT-2 (two LMs whose pretraining data we're pretty sure are not already contaminated), 25 text classification datasets, and across a few settings for the number of training and test observations.

For each dataset, the experiment sanity checks that pretraining on unlabeled text (independent of the test set) helps, i.e., that there's an effect to detect. Call this the pretraining boost. Next, the experiment evaluates the bias from pretraining on unlabeled text from the test set instead of unlabeled independent text. Call this the evaluation bias.

Results

The distributions below are those of marginal effects: averaged across 2 LM types, the 25 classification tasks, and their subsamples.

Pretraining on unlabeled text is, on average, clearly beneficial in every setting. Despite this, there isn't evidence of unfairness. The evaluation bias bounces around 0, and is insignificant. At a task level, the results were consistent: pretraining was beneficial in all but 2 of the 25 tasks, and the evaluation bias was positive in 12 of them, negative in 13 of them, and always less than 1% in absolute value.

Overtraining

An easy way to boost test set performance is to overtrain on the labeled test set. What if we overtrain on the unlabeled test set?

Also, at a task level, there isn't evidence of a statistically significant evaluation bias.

Zero-shot

Let's pretrain Mistral 7B on prompts—instructions followed by the unlabeled text—and then immediately prompt it to do the task. Pretraining is done via QLoRA.

Experiments so far padded sequences. Packing sequences is closer to what's done in the initial pretraining stage of an LLM. Packing allows the model to attend to previous texts, which may result in greater flexibility in modeling unlabeled text. Does packing impact evaluation bias differently than padding?

No.

At a task level, there isn't evidence of a statistically significant evaluation bias.

Meta-analysis

I also wanted to see how (un)stable few-shot learning might be. In the experiment above, this variance is exposed by taking up to 100 subsamples from each dataset. What if the experiment took just one subsample? This is what most few-shot benchmarks effectively do.

It turns out that if we sampled 500 unlabeled texts, 100 classification examples, and 500 evaluation examples for each of the 25 datasets, there's a 47% chance that the experiment would've reported a non-negligible positive or negative bias, instead of the close-to-0 bias we found from repeated subsamples. In a really rough sense, including 25 datasets with 500 observations each seems like a lot of data. In reality, it'd be no better than flipping a coin.

Answer 2

We need more studies and more thinking such as what @chicxulub wrote above. Speaking in general, unsupervised learning (UL) can lead to false comfort in the structures learned by UL (e.g., principal component loadings may change if computed on a new sample). But the way in which UL leads to supervised prediction makes this primarily unrelated to overfitting. Rephrased, we may not always need to include the UL inside a resampling loop that is used to get an unbiased estimate of model performance (and please don't use classification accuracy as a performance measure). But I need to see studies that actually verify that.