Regression - variance of predictions much lower than variance of target

Question

I am using non-negative lasso(sklearn) on a dataset with 1.5MM data points and 120 features. It is a low R2 environment (working with noisy financial data), so $R^2$ is about 10%. What I am more worried though, is that the standard deviation of the predictions is about $\frac{1}{4}$ of the standard deviation of the target variable. Similarly, mean(abs(target)) / mean(abs(predicted)) is about $\frac{1}{4}$.

How can I get that ratio to be closer to 1? I am willing to sacrifice some $R^2$ to achieve this. Do I need to do different type of regression, transform my features in some way, or is there anything else that can be done? In other words, the predictions are too smooth for my application.

If possible, I would like a suggestion how to get results (predictions) that are similar size to target, while still having similar (now much lower) $R^2$.

Maybe I should use different objective function rather than min. sum of squares?

Answer 1

My answer will focus on the baseline OLS case, but the mechanics are similar for techniques like Lasso (although I'll admit that I do not know how $R^2$ is computed for such methods). Also, my answer relates to in-sample fit.

Recall that $R^2$ is defined as (also recall that the mean of the fitted values equals the mean of the $y$, $\bar y=\bar{\hat{y}}$) $$ R^2=\frac{(\hat y-\bar y)'(\hat y-\bar y)}{(y-\bar y)'(y-\bar y)}, $$ which we may rewrite into the ratio of variance explained to variance of the dependent variable, $$ R^2=\frac{\frac{1}{n-1}\sum_i(\hat y_i-\bar y)^2}{\frac{1}{n-1}\sum_i( y_i-\bar y)^2}=\frac{\hat\sigma^2_{\hat y}}{\hat\sigma^2_{y}}, $$ So, when you have a low $R^2$, that is tantamount to saying that the standard deviation of the predictions is less than the standard deviation of the target variable. A fortiori, if you "sacrifice" $R^2$, that ratio can only decrease further.

Here is a little graphical illustration, in which both the $y_i$ (blue) and the fitted values (salmon) are projected onto the y-axis, for a dataset in which $R^2$ is relatively low. We observe that the variation of the fitted values is, as expected, smaller.