Regression results have unexpected upper bound

Question

I try to predict a balance score and tried several different regression methods. One thing I noticed is that the predicted values seem to have some kind of upper bound. That is, the actual balance is in $[0.0, 1.0)$, but my predictions top at about $0.8$. The following plot shows the actual vs. the predicted balance (predicted with linear regression):

And here are two distribution plots of the same data:

Since my predictors are very skewed (user data with power law distribution), I applied a Box-Cox transformation, which changes the results to the following:

Although it changes the distribution of the predictions, there is still that upper bound. So my questions are:

• What are possible reasons for such upper bounds in prediction results?
• How can I fix the predictions to correspond to the distribution of the actual values?

Bonus: Since the distribution after the Box-Cox transformation seems to follow the distributions of the transformed predictors, is it possible that this is directly linked? If so, is there a transformation I can apply, to fit the distribution to the actual values?

Edit: I used a simple linear regression with 5 predictors.

Answer 1

Your dep var is bounded between 0 and 1 and thus OLS is not fully appropriate, I suggest beta regression for instance, and there may be other methods. But secondly, after your box-cox transformation, you say that your predictions are bounded, but your graph doesn't show that.

Answer 2

While there is a lot of focus on using regressions that obey the bounds of 0/1, and this is reasonable (and important!), the specific question of why your LPM does not predict results greater than 0.8 strikes me as a slightly different question.

In either case, there is a noted pattern in your residuals, namely, your linear model fits the upper tail of your distribution poorly. This means there is something nonlinear about the correct model.

Solutions that also consider the 0/1 bound of your data: probit, logit, and beta regression. This bound is critical and must be addressed for your work to be rigorous, given your relatively close to 1 distribution, and thus the large number of answers on that topic.

Usually, though, the problem is that a LPM exceeds the 0/1 bound. This is not the case here! If you are not concerned with the 0/1 bound and actively want a solution that can be fitted with (x'x)^-1(x'y), then consider that perhaps the model is not stictly linear. Fitting the model as a function of x^2, cross products of independent variables, or logs of independent variables can help improve your fit and possibly improve the explanatory power of your model so that it estimates values greater than 0.8.