1
$\begingroup$

The question is more theoretic. Lets assume that the test set is perfect representation of the population.

If I will evaluate the machine learning predictive model on the test set, can we call the good results the absolute proof that the model will work in real word? By good result I mean result that we consider acceptable for us to call the model "correctly working and giving acceptable good results"?

$\endgroup$
1
$\begingroup$

Let's assume that the test set is perfect representation of the population.

This is a really bad assumption because it's hardly ever true. What's happened before isn't necessarily informative about what will happen in the future. Past performance is not indicative of future results.

If I will evaluate the machine learning predictive model on the test set, can we call the good results the absolute proof that the model will work in real word?

No. Even if the test set is drawn from the same distribution as the data that you'll encounter in the future, that data is still a sample. Repeated samples from the same distribution exhibit random variation -- a pattern or measurement you find on the test data might be a statistical artifact.

Of more immediate relevance is the question of overfitting the test data. If I keep training and tuning my models until I get a good result on the test data, there's a good chance that I've obtained a completely bogus result because I've done a lot of work to find a model with a good score on the test data.

$\endgroup$
4
  • $\begingroup$ I agree if we consider that real world is not perfect. But my question is not about real world and it is about the world were we have test set that is perfect representation of the population. Also I am interesting on the prediction of the results not in future were things can change but in the same time when we collected our perfect sample (so there is no changing of the environment). This is theoretical question and not practical real world one. $\endgroup$
    – vasili111
    Mar 5 at 1:04
  • $\begingroup$ The second part of my answer doesn't assume a changing environment, it describes a static distribution and a (poor) practice of model-building. $\endgroup$
    – Sycorax
    Mar 5 at 1:11
  • $\begingroup$ Thank you. I agree with that part too. But in my question I specifically mentioned that test set is perfect representation of the population. I agree that that condition is idealistic and or impossible to meet but my question anyway uses that condition. $\endgroup$
    – vasili111
    Mar 5 at 1:27
  • $\begingroup$ It seems like you're attaching a lot of significance to the idea of a "perfect representation" of a distribution, and this significance is not attached to any technical meaning. I can roll a die $n$ times and compute the average, but that average is unlikely to be exactly the same as the average result from the next $n$ rolls. Likewise, drawing new data from the same distribution is not sufficient to guarantee that it will reproduce the relationships you observed in previous draws. $\endgroup$
    – Sycorax
    Mar 5 at 1:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.