Linear Regression Models: Behaviour

Question

If the target value in the training data is in the range $[5,11]$. Is it possible for a linear regression model to predict absurd values between $[-10, 300]$ (not the exact range but close) in the validation/test data.

I have trained a linear regression model (scikit-learn) on a training data (dimension 2000, training points 3000) with target values in the bounded set as described above but the root mean squared error is in the order of 100s whereas an xgboost/random forest model is having a rmse of less than 1.x. If I just predict the mean of that range ($[5,11]$) i.e. 8 for every validation point, I will still get an rmse of around 3.

Why and how is the linear model behaving that way and how can I analyze the model for improvements? If the dimension of the data is too large, then is it common for linear models to perform that bad or is it due to skewed nature of the features?

Answer 1

There are several issues that i can think of, note that they are not mutually exclusive and they can reinforce eachother:

- Degrees of freedom issues

This is the most important issue, i think. Regressing 2000 variables on 3000 observations gives you about 1.5 observations per variable to use for estimating their effects. One common reference here is Frank Harrell's Regression Modeling Strategies, where the author suggests about 15 observations per variable to get sensible results. So in your case you would need about 10 times more observations.

This could be one of the reasons the tree based methods you mentioned are doing better. They do not necessarily use all variables. Only the ones that result in a split. Looking at the tree-based results, you can see whether only a few variables were used (variable importance) and it can give you clues.

- Collinearity

In a linear regression, you cannot just throw in 2000 variables (either on such a small sample or bigger samples) and expect thing to just work out! How are the correlations between the variables? I am surprised that this worked in the first place. Perhaps sklearn drops collinear variables by default? Even if there is enough information for the estimation to take place, the coefficients will not be correct (i.e., they will be biased) and the results poor. OLS will not be able to attribute the correct effect size to the variables if they are highly correlated

- Distribution of Variables in train and test set and/or outliers

If you split your dataset randomly this shouldn't be a problem, but if not, the coefficients estimated from the training set could be way off the true effects and produce invalid predictions for the test set. Remember OLS is a relatively high variance estimator (compared to e.g., Random Forests) so unseen value ranges could produce weird results. This is where outliers come into play too.

More information on what you have done could lead to a more focused answer. For example, the prediction range you mentioned...is it generally bad, or are there a few extreme errors but the rest is fine? This could point to outliers in the test set. in any case some variable selection/dimentionality reduction seems reasonable before doing OLS given your sample and feature size.