Regression - Poor Outcome from Linear Models that seems easy to improve

Question

I have fit a multiple linear regression on some data using both R and Excel. Both give similar results.

However, for my domain, and in my opinion, the outcome seems to be obviously "off", as shown by the plot of observed vs predicted below:

For my problem domain, getting the large values "right" is more important than getting the large set of smaller values right. So I just did a quick adjustment of my predicted values by multiplying the predicted values from my regression with correction factors based on the observed value, in which larger values have larger correction weights.

To do this I just did a lookup of the correction factor based on the observed value. For example, for all observed values in the range 0 - 0.2 the predicted values were multiplied by 1 (no adjustment), for values in the range 0.2-0.6 they were multiplied by 1.2, etc. until the large values, range 1-3, were multiplied by 2, etc.

I then used Excel's solver to find the optimal values for these correction factors, minimizing the sum of squared residuals. This gave me a much better solution for my problem, as shown below:

I thought my hacked solution would violate certain principles of Linear Models but to my surprise when I checked the sum of squared residuals on my adjusted outcome, they were significantly lower than the initial outcome of the MLR model.

My question is: Why does the Multiple Linear Regression model not automatically search and find the second solution if the residuals are lower?

BTW, my model has 4 predictor variables, and there are no log transformations involved.

Answer 1

Why does the Multiple Linear Regression model not automatically search and find the second solution if the residuals are lower?

Because the model you've implemented is not a linear model.

If you were to apply a single correction factor irrespective of the size of the predicted variable, then this is equivalent to a linear model because you are just scaling the coefficients.

But, because your correction factor is a function of the prediction, then this is no longer in the space of linear models and hence can not be estimated by linear regression.

If you need a more flexible model, I think multivariate adaptive regression splines might be a good next step for you. If I've understood your approach correctly, MARS should be a reasonable facsimile to your reweighing approach.