# Why does my OLS model produce high error on test set?

To create the feel of using my model to predict unknown data, I split the dataset I have into training set and test set. The $$R^2$$ value of my OLS regression model in the training set is decent (89.16%), the residual does not violate assumptions, and the MSE is considerably small (0.0151 for response with range 0-13). When using the model to predict the test set, however, I get very high MSE (158.64). My model does not include polynomial terms at all, just linear (degree one), if it matters.

My questions:

1. Is this normal to happen? I believe this is not a case of overfitting (I believe overfitting just does not happen in OLS).

2. How can this happen? Is there a pattern in the test set that is completely different from the pattern learned in the training set, therefore the high MSE?

3. Other than creating other types of models, is there anything I can do regarding this? Is it my model or the data that makes trouble?

• 1) How many parameters are in your OLS model? 2) What are the train and test MSEs for the intercept-only model? (Take all train and test predictions to be the mean of the training group. What are the MSEs?)
– Dave
Commented Dec 9, 2019 at 12:46
• Overfitting can definitely happen in OLS. Commented Dec 9, 2019 at 13:01
• If your in-sample MSE is 0.0151 for a target variable that is between 0 and 13, you either have the most well-behaved dataset I have ever seen, or you are severely overfitting. Without having seen your data, my money is on the latter. (Why would overfitting "just not happen in OLS"?) Commented Dec 9, 2019 at 13:35
• Because I don't see a way where overfitting can happen in OLS, because OLS simply get one set of coefficients after a single calculation, unlike methods where iterations could go way down to actually detect noises - correct me if I'm wrong, In the case overfitting could happen in OLS, how? Commented Dec 9, 2019 at 17:08
• @Dave 1) An intercept, two parameters for continuous variables, and 27 parameters for dummy variables. 2) I'm beginning to see a problem because using training prediction against training mean, the MSE is 0.1398 but using test prediction against test mean, the MSE is 77.3041. Commented Dec 9, 2019 at 17:16

Two likely options are:

• overfitting: you might have too many variables/interactions/high coefficients. Try inserting a regularization via Ridge or Lasso regression to reduce the number of regressors and the magnitude of the coefficients
• wrong split: if you did not shuffle the data before splitting, there might be some ordinal/time related component that you did not take into account. Are features distributed in a similar way in train and test datasets? Can you shuffle your observations before you split?

Try address the above points, it might help find a solution. Hope it helps!

• I already picked my test set randomly, and I believe it's equivalent to shuffling as you suggest. Commented Dec 9, 2019 at 17:19