Zero Residuals in the Test Data

Zero (or very small) residuals in the test data are an indication that the model is very accurate. Does it make sense? Any caveats to that?

UPDATE

The model was fit on a training set and the issue discussed is for the test set. Both sets were scaled. I've performed a residual analysis using both graphical (e.g., residual vs. predicted, histogram, Q-Q plot) and numerical (Kolmogorov-Smirnov, Anderson-Darling, Shapiro-Wilk tests) approaches. But I'm wondering if having zero residuals on the test set is an indication of a problem (e.g., modeling assumption), or not.

That wholly depends on the model's purpose and your notion of accuracy:

Small/zero residuals are fine, if you are just interested in fitting a line through the test data points (e.g. using splines).

They are bad, if you want to make predictions based on the model in a random test setting, as they suggest heavy overfitting. In this case, you are much better off with looking at the "residuals vs. fits" plot, in order to look for potential problems with the model (outliers, patterns, non-symmetry around zero, etc.).

EDIT: As you used a (different) training set for fitting the model, things are a little bit different:

• Overfitting is obviously not an issue here, assuming the sets were independently drawn. But still, the absolute size of the residuals by itself doesn't say much, as was pointed out by chRrr - if the variance is very small, the residuals will naturally be smaller, too. Hence, they are affected by scaling.

• When comparing different models, fitted on the same training sample, a smaller $RSS$ (meaning "smaller residuals") on the test sample certainly is preferable.

• If you are fitting a linear model, you can also go with the $R^2$, in order to link the size of the residuals ($RSS$) with the test data variation.

In summary: The absolute size of test data residuals alone doesn't say anything about model accuracy. However, it can be used to assess the quality of a model, when comparing it relative to other values, like the test data variation (for linear models) or other models' residual size.

• as Bruno explicitly wrote that he measured the performance on a test sample (implying, that he learned the model on a training set), I wonder if overfitting can still occur here, provided the test set is different from the training set. Nevertheless very small residuals might also still be derived as the effect of some kind of scaling (i.e. they depend on the units of your variables). Aug 15 '17 at 15:19