Lasso regression prediction on test set is predicting towards the mean of the train set?

Question

I am using lasso regression to predict age (continuous data) from a set having 2112 numeric features (indepedent variable).

The training dataset contains around 2773 participants. The mean of that dataset's outcome variable is 62.6 and the mean of the predicted age is also around the same 62.4. I have used gridsearchCV for hyper parameter tuning.

I am using this trained model on a number of test datasets. The means of the test dataset's outcome variable ranges from 61.6 to around 68.87.

However, for all of these test datasets, the mean of the predicted value converges to around 62.6 (which almost corresponds to the mean of the train dataset).

Is my model overfitting on the train dataset and if so how do I prevent this from happening?

Answer 1

As an initial guess, overfitting the test data set probably isn't your problem.

For linear models, Statistical Learning with Sparsity (SLS) notes on page 18:

Somewhat miraculously, one can show that for the lasso, with a fixed penalty parameter $\lambda$, the number of nonzero coefficients $k_{\lambda}$ is an unbiased estimate of the degrees of freedom

Your comment indicates that you had 388 nonzero coefficients for 2773 observations. That's about 7 observations per degree of freedom (df). Usual rules of thumb for linear regressions and continuous outcomes suggest that you can avoid overfitting if you have 10-20 cases per df that you use up. So there might be some overfitting, but it doesn't seem enough to explain the results you describe on test data.

To test overfitting of your LASSO fits on training data, you can use bootstrapping. SLS describes how to use that properly for LASSO in Section 6.2. Overfitting of the training set can be evaluated with the optimism bootstrap, in which you repeat the modeling process on multiple bootstrap samples and evaluate the difference in performance of each model between its bootstrap sample and the full training set.

Ridge regression, which keeps all of the predictors but penalizes their coefficients, might work much better. LASSO can work well when only a small subset of predictors are strongly associated with outcome and there aren't other predictors correlated with them. If this is brain imaging or similar data, however, I suspect that there are massive correlations among your 2112 features and that each individually only has a small association with outcome. Try ridge regression, and evaluate its internal performance on the training set as suggested above for LASSO.

I suspect, however, that your problem has more to do with omitted-variable bias; from one of your comments:

the datasets are comparable in terms of age, sex etc but not on the presence of the disease as such.

In linear regression, omitting a predictor that is both correlated with outcome and with included predictors will lead to incorrect assessment of regression coefficients. It sounds like "presence of the disease as such" has those characteristics and isn't included in your model. In that case, your results on test sets might not be so surprising.