I am currently doing my college final project. I forecasted national soybeans yield and used MAPE to calculate the in-sample and out-of-sample forecasting accuracy. The MAPE results showed that the in-sample forecasting accuracy is higher than the out-of-sample accuracy. Does in-sample forecasting results always supposed to have a higher accuracy than the out-of-sample forecasting accuracy? and why is that? I can not seem to find the explanation in my text book, so if you happen to know about in-sample and out-of-sample forecasting accuracy, please help me >< thank you so much!^^
1 Answer
$\begingroup$
$\endgroup$
3
In-sample accuracy is almost always higher than out-of-sample accuracy, simply because in-sample is the data you are fitting your model to. So this is no reason for concern. (To the contrary, if your out-of-sample accuracy were higher than in-sample, I would wonder whether something is broken.)
This is part of the reason why in-sample accuracy is very rarely reported on. It's not a good guide to out-of-sample accuracy, and optimizing in-sample accuracy can lead to overfitting.
You may find Forecasting: Principles and Practice (2nd ed.) by Athanasopoulos & Hyndman and Forecasting: Principles and Practice (3rd ed.) by Athanasopoulos & Hyndman interesting reading.
answered Jun 28, 2021 at 13:16
-
$\begingroup$ Thank you so much for answering my question^^ But, I am still having trouble understanding your explaination on part 'simply because in-sample is the data you are fitting your model to'. If you don't mind, can you explain more about this? And if the out-of-sample accuracy were higher, does it means that the model is overfitting? Thank you! $\endgroup$– adinCommented Jun 29, 2021 at 11:28
-
$\begingroup$ When you train (or fit, whatever term you prefer) your model, you are fitting it to the training data, and you are explicitly aiming for a good fit. Conversely, you don't fit it to the (unseen) testing data. So your fitting will naturally lead to a higher accuracy in-sample than out-of-sample. To your second point, overfitting results in much lower accuracy out-of-sample than in-sample. Higher accuracy out-of-sample than in-sample will usually only occur if you have data leakage. $\endgroup$ Commented Jun 29, 2021 at 11:53
-
$\begingroup$ Thank you so much for your explanation, it really helps a lot! I hope you have a good day and stay safe!^^ $\endgroup$– adinCommented Jun 29, 2021 at 13:11