Is it valid to change the model after seeing the results of test data?

Question

I have seen this question, which says using test data to tune hyper-parameters is not a good decision and would stops you from generalizing the model.

My question is different from the above question, I know that observing the test data is not a good work.

Question: Is it a valid work to change the model after evaluating it with the former model and get a better result or not? why?

For example you have developed linear model but after evaluating it with test data, change it to polynomial with second degree and report the results (which is better than linear).

Answer 1

If you do that, then your "test" data is no longer entirely test data --- it is now partly training data. Indeed, the entire distinction between these two classes is that the training data is used to formulate hypotheses and models, and the test data is then used to make inferences in relation to those hypotheses under the model. For that reason, it is a good idea to take care to formulate a sufficiently general model at the training stage, noting that you might want to include higher-order effects that allow for more general functional forms than are exhibited in the training data.

In any case, if you use your "test" data to adjust the model, but then still treat it as test data, you are effectively using the data twice, first as training data, then as test data. The danger here is that the model choice might relate to the hypotheses of interest, in which case this method induces confirmation bias in your analysis, whereby the tests are biased in favour of acceptance of the hypotheses. If your modelling changes are unrelated to the hypotheses of interest then you might get away with it without imposing bias (or at least, without imposing too much bias), but it is very difficult to be sure. If you decide to do this, I would suggest that you at least conduct a sensitivity analysis that compares the conclusions from your preferred model with the conclusions from the original model. That way you can check to see if the change in the model affected any of the conclusions in relation to the hypotheses of interest (or if it just fit better in other respects).

Answer 2

One way to solve your problem is to just use Cross-validation across the whole set, as opposed to a normal test-train split. The theoretical details of how this works are explained in Chapter 7 of ESL. Intuitively a test-train split is just a 1-fold cross-validation, so, sample size permitting, you can try a k-fold cross-validation instead.

Ch. 7 of ESL also mentions other possibilities such as using information criteria (AIC, BIC, etc...) or structural risk minimization as proxies for out-of-sample model performance.

There is a more interesting "big picture" question implicit in your post though. If you are going to use your model for a real world problem, it is more than likely that your production data distribution will eventually drift, and the challenge of re-estimating your model accordingly is part of the much wider topic of ML Ops, which I definitely recommend you look at if you are interested in applying DS and ML to real world use cases.

Answer 3

The process you describe is not appropriate. By using the test set twice, you are allowing that data to influence your model selection. That data thus becomes part of your modeling process rather than serving as a means of external validation.

Answer 4

If you did so, you added "additional information" in your training process and that would incur over-parameterization issues in your final model.
Suppose you are performing supervised learning techniques for your tasks, from the empirical risk minimization perspective, we try to look for a function $h$ that minimizes the loss between prediction $\hat{y}$ and true outcome $y$, in other words, we look for a function that best represents the map $X\to Y$. The reason why we need more data to better approximate our model is that CLT guarantees as $n \to \infty$, the risk will converge to 0, which laid the theoretical foundation for the reason why we need more examples to train our model in supervised learning problems.
Back to your example, what if a polynomial function fits better than the linear model? I assume this discrepancy comes from two possible situations:
1 the class distribution is severally imbalanced.
2 the training data isn't representative of the entire data set.
In order to handle these issues, you can obviously try cross-validation procedures in order to get an overall sense of entire data distribution. In addition, oversampling or undersampling methods are alternatives for dealing with class-imbalanced issues. But test data should only be used when a model is already trained, otherwise, it shall be independent of any steps beforehand. Hope this answer helps.