Does overfit outside the dataset range in Machine learning really matter?

Question

I was tyring out a pizza price dataset. With two different approaches, the results were different (as shown in the screenshot below). As the dashed line shows, the model was overfitted. But it is clear that it actually started to overfit after the biggest available pizza size. And within the reasonable pizza size range, the dashed line fits way better than the straight line.

I've encountered this situation several times with different datasets. So I wonder that when we limit the data parameter range, does it really matter that the model overfits outside of the range?

Answer 1

Yes, it still matters because you are biasing your predictions based on the training data. In your graph, you can see that the model forecasts a higher value for the values in the middle and lower for the values on the end. This means, that if there is actually a linear relationship between the diameter and the price, you will see higher errors due to this bias.

If on the other hand, the relationship isn't actually linear, it may be ok to have a graph like this. If you do have a relationship between diameter and price where it increases quickly for the low values and then levels off as the diameter gets larger, you would want to capture that in the model. A logarithmic model might be a better way of doing this, but if for one reason or another that isn't a reasonable model for you to build, you might be able to approximate the behavior in the value range of interest using a quadratic model.

The important thing is to try to capture the relationship between your predictors and your outcome with the acknowledgement that generally simpler models can do that better in small data. If you know your data don't behave the way your model does, you are unlikely to get good results.