Overestimating the lower values and underestimating the higher values in Regression

Question

I've tried many different models from Linear to non-linear and flexible models like random forest to solve a regression problem. But all of them apparently overestimate the lower values of the response variable and underestimate the higher values. Below you can see the "observed versus fitted" plot of the random forest model:

Question: What are the possible reasons behind this issue and how it be fixed?

Answer 1

This is just an effect of a model which only partially explains the observations. In such a case, this sort of effect is common. Possible choices include accepting that some things are random or trying to find better explanations for the observations; it may be impossible to get a better result with the data you have.

Here is an illustration of a linear model with a correlation between the observed variable and the explanatory variable of about $0.5$ (in some fields this is high, and in others low). The second chart is similar to yours. Generating some simulation data in R:

set.seed(2023)
X <- rnorm(1000)
Observed <- X*0.5 + rnorm(1000)*0.9
cor(X,Observed)
# 0.4996774
plot(X, Observed)
fit <- lm(Observed ~ X)
abline(fit)

you get a linear regression which looks like this, and you can see the positive correlation and noise. Nothing surprising here.

Now drawing the equivalent of your chart, you again get the obvious gap between the points and the blue diagonal. This is what happens with regression (and indeed is part of the etymology)

plot(Observed, fit$fitted.values, ylim=c(min(Observed),max(Observed)))
abline(0,1,col="blue")

This is in effect the same chart as previously with a reflection and some stretching and relocation. The point at the bottom the first chart has become the point at the far left of the second.

A better chart for seeing if there might be something wrong with the model, beyond a lot of noise, would be to plot residuals against fitted values and see whether there is a obvious pattern which could be addressed with a better model using the existing data. In this case there is no such obvious pattern. (The point I suggested looking at on the previous charts has now moved to the top and looks less special.)

plot(fit$fitted.values, fit$residuals)

Answer 2

I suspect that there is not a very strong "signal" in your data, meaning that for a given place in x-space there is a big variance in $y$, and therefore $y$ cannot be predicted very precisely. The implication is then that where big values of $y$ ("observed") occur, this is often not because the corresponding x-values that you use to "explain" the $y$ enforce a large $y$, but rather it can happen all over the place because of noise/variation. So it is not possible to predict/fit the biggest values well, and neither the smallest.

If you have already tried out much and this happens all the time, chances are this is how it is and you have to accept it. In some problems the x-values do not work very well as predictors.

Answer 3

[Edited as I misread the original question]

The way a Random Forest works, simplified, is that the computer tries to sort the observations into buckets, then calculates the average value within each bucket, and uses that as the score for everything in that bucket. This is done repeatedly, getting different bucketing rules, and then the final score for an observation is the average of the scores for all the buckets it was in.

One issue with Random Forest is that it can "top out": a set of observations can be sorted into the "highest bucket" (or the "lowest bucket"), and from there, no further distinction is made within the set. If the same set is topping out for each bucketing rule, then you're not going to be getting a different score for them.

Something along these lines appears to be happening with your predictions. Your model isn't predicting less than -1, or much more than 1, which suggests that the set all observations with a true value greater than 1 is topping out, and vice versa for less than -1.

Now, your data also seems to be much more sparse for those data points, so your model is probably prioritizing the more central ones. Furthermore, you may not have enough datapoints, regardless of what you do, to make predictions for those observations. Or the explanatory variable may just not be explanatory anymore at the extremes (for instance, age is a good predictor of height for schoolchildren, but not so much for adults; if you have that sort of relationship, there might not be anything you can do). But things you can try are:

1. Create separate models for the extremes
   
2. Look at your extremes and try to find distinguishing characteristics
   
3. Generate new features, such as doing a linear regression on the original features, and feeding the output of that into the Random Forest model as an input.