Timeline for Is an overfitted model necessarily useless?
Current License: CC BY-SA 3.0
10 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| May 11, 2017 at 13:50 | history | edited | Metariat | CC BY-SA 3.0 |
added 299 characters in body
|
| May 11, 2017 at 13:20 | comment | added | Richard Hardy | But to be fair, when I re-read your answer for the second and the third time, it is getting clearer what you mean. My comments then are about making it even better and even clearer. | |
| May 11, 2017 at 13:17 | comment | added | Richard Hardy | @Metariat, you are right that the test error is an estimate and it could be different from one test set to another. However, as I mentioned before, there is no reason to expect that the test error underestimates the true error (it does not, on average). So by taking a large-enough test sample, we can bound the test error with a desired level of confidence at a desired range. Now more practically, perhaps you should define the test error by editing your answer to make sure there is no misunderstanding of what you mean when contrasting the test error with the out-of-sample error. | |
| May 11, 2017 at 13:08 | comment | added | Metariat | From my perspective, testing error is the error obtained when applying the model into a blinded set, it is an approximation of the out-of-sample error, which is the error obtained when applying the model into the whole population. And it is not the same, the valuable information is the out-of-sample error. And when the model is overfitted, the testing error is not stable, and bad supprises could happen on the other data sets. | |
| May 11, 2017 at 12:49 | comment | added | Nuclear Hoagie | Can you elaborate on the difference between out-of-sample error and testing error? From my understanding, both are the error found when applying the model to samples not used to train the model. The only possible difference I can see is when using time-series data, the out-of-sample data should be from later time points, but this questions makes no mention of that. | |
| May 11, 2017 at 12:49 | comment | added | Richard Hardy | I think you might have confused the validation set and the test set, and their respective errors. The test error is the out-of-sample error. While validation error is an optimistic measure of a selected model, test error is not. The test error is an unbiased estimate of how the model will perform on a new sample from the same population. We can estimate the variance of the test error, so we are quite fine by knowing only the test error as long as the test set is not too small. @Hossein | |
| May 11, 2017 at 12:44 | comment | added | Hossein | I just modified my comment. Could you please take a look at it? You say that the difference between the training error and the test error is an indication of the variability of the test error, but I think the test error can be robust because it is not seen during training. | |
| May 11, 2017 at 12:42 | comment | added | Metariat | yes, it could perform 70% accurate in the real world, even better, but could be worse. The problem with this approximation is that the 70% obtained is highly variable, and we don't know how variable it is, in contrary to the "optimal" model. | |
| May 11, 2017 at 12:38 | comment | added | Hossein | Since the used test set that obtains 70% accuracy is not seen in the training phase, is not it a good estimation of the out-of-sample error? I think the difference between training error (100%) and testing error (70%) is not a good indication of the difference between out-of-sample error and test error. It is possible that the overfitted model performs 70% accurate in the real world, while it is 100% accurate for the training data. I expect training error to be lower than test error, since the training data are used to generate the model, but the test data are not seen during training. | |
| May 11, 2017 at 12:27 | history | answered | Metariat | CC BY-SA 3.0 |