# Is it valid to change the model after seeing the results of test data?

I have seen this question, which says using test data to tune hyper-parameters is not a good decision and would stops you from generalizing the model.

My question is different from the above question, I know that observing the test data is not a good work.

Question: Is it a valid work to change the model after evaluating it with the former model and get a better result or not? why?

For example you have developed linear model but after evaluating it with test data, change it to polynomial with second degree and report the results (which is better than linear).

• I’m not sure how this is different. Could you provide some more detail? – Demetri Pananos Jul 11 at 23:05
• @DemetriPananos the difference is where in first problem (link) it tunes the hyper-parameters that would destroy generalization. (Not overfitting in every situation) but in the latter problem, it uses completely different model, such as polynomial rather than linear. – Amir Hooshang Jul 11 at 23:22
• So you would for example fit a linear model, say “meh I don’t like this”, change to a different model, and then evaluate on the test set? Have I got that right? – Demetri Pananos Jul 11 at 23:47
• And so after changing the model, how would I determine in that model is appropriate or not? – Demetri Pananos Jul 11 at 23:52
• @MostafaGhadimi evaluating it on the same test set afterwards yields some numbers that are almost worthless and can not be used to provide a reasonable evaluation estimete. If you want to know whether the changed model is appropriate or not, you would need new data, as you've made the previous test set useless for obtaining a realistic evaluation - you can only use it to get an "optimistic" relation that's likely to be "better" than the actual model performance by some unknown, unknowable amount. – Peteris Jul 12 at 19:47

If you do that, then your "test" data is no longer entirely test data --- it is now partly training data. Indeed, the entire distinction between these two classes is that the training data is used to formulate hypotheses and models, and the test data is then used to make inferences in relation to those hypotheses under the model. For that reason, it is a good idea to take care to formulate a sufficiently general model at the training stage, noting that you might want to include higher-order effects that allow for more general functional forms than are exhibited in the training data.

In any case, if you use your "test" data to adjust the model, but then still treat it as test data, you are effectively using the data twice, first as training data, then as test data. The danger here is that the model choice might relate to the hypotheses of interest, in which case this method induces confirmation bias in your analysis, whereby the tests are biased in favour of acceptance of the hypotheses. If your modelling changes are unrelated to the hypotheses of interest then you might get away with it without imposing bias (or at least, without imposing too much bias), but it is very difficult to be sure. If you decide to do this, I would suggest that you at least conduct a sensitivity analysis that compares the conclusions from your preferred model with the conclusions from the original model. That way you can check to see if the change in the model affected any of the conclusions in relation to the hypotheses of interest (or if it just fit better in other respects).

One way to solve your problem is to just use Cross-validation across the whole set, as opposed to a normal test-train split. The theoretical details of how this works are explained in Chapter 7 of ESL. Intuitively a test-train split is just a 1-fold cross-validation, so, sample size permitting, you can try a k-fold cross-validation instead.

Ch. 7 of ESL also mentions other possibilities such as using information criteria (AIC, BIC, etc...) or structural risk minimization as proxies for out-of-sample model performance.

There is a more interesting "big picture" question implicit in your post though. If you are going to use your model for a real world problem, it is more than likely that your production data distribution will eventually drift, and the challenge of re-estimating your model accordingly is part of the much wider topic of ML Ops, which I definitely recommend you look at if you are interested in applying DS and ML to real world use cases.

• good luck finding one paper in medical literature that doesn't use an untouched holdout set. – jiggunjer Jul 13 at 15:07
• I worked in retail demand forecasting for a long time. Production demand forecasting systems don't bother with a hold out set at all. – Skander H. Jul 13 at 16:10
• One practice I've seen is to use a holdout set for validation during model development then fit the production model on the entire dataset. – Neal Jul 13 at 17:16

The process you describe is not appropriate. By using the test set twice, you are allowing that data to influence your model selection. That data thus becomes part of your modeling process rather than serving as a means of external validation.

• This is clearly true, but how to actually get around the problem? If you have a limited amount of data, you have to use that to get information about what model is working and which ones are not. And clearly using the test data only as way to refine and improve a model is a lot better than tuning directly on it, even though of course the effect is hard to quantify – Ant Jul 12 at 19:47
• @Ant I'm not sure I understand what you think the "problem" is. OP's approach results in the test set not being a good measure of external validation. There are ways to use the entire data set to estimate out of sample performance (see the Efron Gong Bootstrap for estimating optimism). Are you saying that we have to use the test set for model selection? – Demetri Pananos Jul 12 at 20:03
Suppose you are performing supervised learning techniques for your tasks, from the empirical risk minimization perspective, we try to look for a function $$h$$ that minimizes the loss between prediction $$\hat{y}$$ and true outcome $$y$$, in other words, we look for a function that best represents the map $$X\to Y$$. The reason why we need more data to better approximate our model is that CLT guarantees as $$n \to \infty$$, the risk will converge to 0, which laid the theoretical foundation for the reason why we need more examples to train our model in supervised learning problems.