10
$\begingroup$

I have seen this question, which says using test data to tune hyper-parameters is not a good decision and would stops you from generalizing the model.

My question is different from the above question, I know that observing the test data is not a good work.

Question: Is it a valid work to change the model after evaluating it with the former model and get a better result or not? why?

For example you have developed linear model but after evaluating it with test data, change it to polynomial with second degree and report the results (which is better than linear).

$\endgroup$
  • 4
    $\begingroup$ I’m not sure how this is different. Could you provide some more detail? $\endgroup$ – Demetri Pananos Jul 11 at 23:05
  • $\begingroup$ @DemetriPananos the difference is where in first problem (link) it tunes the hyper-parameters that would destroy generalization. (Not overfitting in every situation) but in the latter problem, it uses completely different model, such as polynomial rather than linear. $\endgroup$ – Amir Hooshang Jul 11 at 23:22
  • $\begingroup$ So you would for example fit a linear model, say “meh I don’t like this”, change to a different model, and then evaluate on the test set? Have I got that right? $\endgroup$ – Demetri Pananos Jul 11 at 23:47
  • 2
    $\begingroup$ And so after changing the model, how would I determine in that model is appropriate or not? $\endgroup$ – Demetri Pananos Jul 11 at 23:52
  • 2
    $\begingroup$ @MostafaGhadimi evaluating it on the same test set afterwards yields some numbers that are almost worthless and can not be used to provide a reasonable evaluation estimete. If you want to know whether the changed model is appropriate or not, you would need new data, as you've made the previous test set useless for obtaining a realistic evaluation - you can only use it to get an "optimistic" relation that's likely to be "better" than the actual model performance by some unknown, unknowable amount. $\endgroup$ – Peteris Jul 12 at 19:47
20
$\begingroup$

If you do that, then your "test" data is no longer entirely test data --- it is now partly training data. Indeed, the entire distinction between these two classes is that the training data is used to formulate hypotheses and models, and the test data is then used to make inferences in relation to those hypotheses under the model. For that reason, it is a good idea to take care to formulate a sufficiently general model at the training stage, noting that you might want to include higher-order effects that allow for more general functional forms than are exhibited in the training data.

In any case, if you use your "test" data to adjust the model, but then still treat it as test data, you are effectively using the data twice, first as training data, then as test data. The danger here is that the model choice might relate to the hypotheses of interest, in which case this method induces confirmation bias in your analysis, whereby the tests are biased in favour of acceptance of the hypotheses. If your modelling changes are unrelated to the hypotheses of interest then you might get away with it without imposing bias (or at least, without imposing too much bias), but it is very difficult to be sure. If you decide to do this, I would suggest that you at least conduct a sensitivity analysis that compares the conclusions from your preferred model with the conclusions from the original model. That way you can check to see if the change in the model affected any of the conclusions in relation to the hypotheses of interest (or if it just fit better in other respects).

| cite | improve this answer | |
$\endgroup$
11
$\begingroup$

One way to solve your problem is to just use Cross-validation across the whole set, as opposed to a normal test-train split. The theoretical details of how this works are explained in Chapter 7 of ESL. Intuitively a test-train split is just a 1-fold cross-validation, so, sample size permitting, you can try a k-fold cross-validation instead.

Ch. 7 of ESL also mentions other possibilities such as using information criteria (AIC, BIC, etc...) or structural risk minimization as proxies for out-of-sample model performance.

There is a more interesting "big picture" question implicit in your post though. If you are going to use your model for a real world problem, it is more than likely that your production data distribution will eventually drift, and the challenge of re-estimating your model accordingly is part of the much wider topic of ML Ops, which I definitely recommend you look at if you are interested in applying DS and ML to real world use cases.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ good luck finding one paper in medical literature that doesn't use an untouched holdout set. $\endgroup$ – jiggunjer Jul 13 at 15:07
  • $\begingroup$ I worked in retail demand forecasting for a long time. Production demand forecasting systems don't bother with a hold out set at all. $\endgroup$ – Skander H. Jul 13 at 16:10
  • $\begingroup$ One practice I've seen is to use a holdout set for validation during model development then fit the production model on the entire dataset. $\endgroup$ – Neal Jul 13 at 17:16
8
$\begingroup$

The process you describe is not appropriate. By using the test set twice, you are allowing that data to influence your model selection. That data thus becomes part of your modeling process rather than serving as a means of external validation.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ This is clearly true, but how to actually get around the problem? If you have a limited amount of data, you have to use that to get information about what model is working and which ones are not. And clearly using the test data only as way to refine and improve a model is a lot better than tuning directly on it, even though of course the effect is hard to quantify $\endgroup$ – Ant Jul 12 at 19:47
  • 2
    $\begingroup$ @Ant I'm not sure I understand what you think the "problem" is. OP's approach results in the test set not being a good measure of external validation. There are ways to use the entire data set to estimate out of sample performance (see the Efron Gong Bootstrap for estimating optimism). Are you saying that we have to use the test set for model selection? $\endgroup$ – Demetri Pananos Jul 12 at 20:03
  • $\begingroup$ Are you rfering to this article? tyigit.bilkent.edu.tr/gradmetr/bootstrap.pdf $\endgroup$ – vasili111 Jul 13 at 16:05
  • $\begingroup$ @vasili111 I believe that is the article yes. There have since been better explanations of the procedure. See Clinical Prediction Models by Ewout Steyerberg $\endgroup$ – Demetri Pananos Jul 13 at 16:24
1
$\begingroup$

If you did so, you added "additional information" in your training process and that would incur over-parameterization issues in your final model.
Suppose you are performing supervised learning techniques for your tasks, from the empirical risk minimization perspective, we try to look for a function $h$ that minimizes the loss between prediction $\hat{y}$ and true outcome $y$, in other words, we look for a function that best represents the map $X\to Y$. The reason why we need more data to better approximate our model is that CLT guarantees as $n \to \infty$, the risk will converge to 0, which laid the theoretical foundation for the reason why we need more examples to train our model in supervised learning problems.
Back to your example, what if a polynomial function fits better than the linear model? I assume this discrepancy comes from two possible situations:
1 the class distribution is severally imbalanced.
2 the training data isn't representative of the entire data set.
In order to handle these issues, you can obviously try cross-validation procedures in order to get an overall sense of entire data distribution. In addition, oversampling or undersampling methods are alternatives for dealing with class-imbalanced issues. But test data should only be used when a model is already trained, otherwise, it shall be independent of any steps beforehand. Hope this answer helps.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.