# Regression - variance of predictions much lower than variance of target

I am using non-negative lasso(sklearn) on a dataset with 1.5MM data points and 120 features. It is a low R2 environment (working with noisy financial data), so $$R^2$$ is about 10%. What I am more worried though, is that the standard deviation of the predictions is about $$\frac{1}{4}$$ of the standard deviation of the target variable. Similarly, mean(abs(target)) / mean(abs(predicted)) is about $$\frac{1}{4}$$.

How can I get that ratio to be closer to 1? I am willing to sacrifice some $$R^2$$ to achieve this. Do I need to do different type of regression, transform my features in some way, or is there anything else that can be done? In other words, the predictions are too smooth for my application.

If possible, I would like a suggestion how to get results (predictions) that are similar size to target, while still having similar (now much lower) $$R^2$$.

Maybe I should use different objective function rather than min. sum of squares?

• Possible duplicate of Regression predictions show far less variance than expected – Stephan Kolassa Feb 7 '19 at 15:13
• I get the sense (from low reputation) that this is a relatively new user. There is a decent chance that they aren't expert at finding previous questions like theirs. This could easily, and probably is, and we should presume it probably is, a genuine question from a novel asker. If that is the case, we should do some hand-holding, point out options, or ask for clarification in areas that might help show the difference between the two questions. – EngrStudent Feb 7 '19 at 16:05
• Thanks, but I believe I've sufficiently elaborated on my question. It says specifically what I am interested in, and the other question does not have any input that addresses my concerns. – RedBaron Feb 7 '19 at 16:15
• @RedBaron, I tried to give an answer to your question, but the linked answer indeed explains the underlying issue also affecting your issue, even if the estimator applied is different. – Christoph Hanck Feb 8 '19 at 10:39
• @gung - I agree. Duplicates are totally cool. Just saying that goes a long way. "It looks like you are new. Yours looks like this one -insert link here-". That is hand-holding. I saw several close votes and only one comment, and that means that folks were doing nearly the opposite of that. – EngrStudent Feb 8 '19 at 17:25

My answer will focus on the baseline OLS case, but the mechanics are similar for techniques like Lasso (although I'll admit that I do not know how $$R^2$$ is computed for such methods). Also, my answer relates to in-sample fit.
Recall that $$R^2$$ is defined as (also recall that the mean of the fitted values equals the mean of the $$y$$, $$\bar y=\bar{\hat{y}}$$) $$R^2=\frac{(\hat y-\bar y)'(\hat y-\bar y)}{(y-\bar y)'(y-\bar y)},$$ which we may rewrite into the ratio of variance explained to variance of the dependent variable, $$R^2=\frac{\frac{1}{n-1}\sum_i(\hat y_i-\bar y)^2}{\frac{1}{n-1}\sum_i( y_i-\bar y)^2}=\frac{\hat\sigma^2_{\hat y}}{\hat\sigma^2_{y}},$$ So, when you have a low $$R^2$$, that is tantamount to saying that the standard deviation of the predictions is less than the standard deviation of the target variable. A fortiori, if you "sacrifice" $$R^2$$, that ratio can only decrease further.
Here is a little graphical illustration, in which both the $$y_i$$ (blue) and the fitted values (salmon) are projected onto the y-axis, for a dataset in which $$R^2$$ is relatively low. We observe that the variation of the fitted values is, as expected, smaller. 