0
$\begingroup$

I have fit a multiple linear regression on some data using both R and Excel. Both give similar results.

However, for my domain, and in my opinion, the outcome seems to be obviously "off", as shown by the plot of observed vs predicted below:

enter image description here

For my problem domain, getting the large values "right" is more important than getting the large set of smaller values right. So I just did a quick adjustment of my predicted values by multiplying the predicted values from my regression with correction factors based on the observed value, in which larger values have larger correction weights.

To do this I just did a lookup of the correction factor based on the observed value. For example, for all observed values in the range 0 - 0.2 the predicted values were multiplied by 1 (no adjustment), for values in the range 0.2-0.6 they were multiplied by 1.2, etc. until the large values, range 1-3, were multiplied by 2, etc.

I then used Excel's solver to find the optimal values for these correction factors, minimizing the sum of squared residuals. This gave me a much better solution for my problem, as shown below:

enter image description here

I thought my hacked solution would violate certain principles of Linear Models but to my surprise when I checked the sum of squared residuals on my adjusted outcome, they were significantly lower than the initial outcome of the MLR model.

My question is: Why does the Multiple Linear Regression model not automatically search and find the second solution if the residuals are lower?

BTW, my model has 4 predictor variables, and there are no log transformations involved.

$\endgroup$
  • 1
    $\begingroup$ The relationship is not linear over the entire range. $\endgroup$ – Glen_b -Reinstate Monica Sep 6 '19 at 1:47
  • $\begingroup$ "For my problem domain, getting the large values "right" is more important than getting the large set of smaller values right." That seems like an argument to simply weight observations. $\endgroup$ – Roland Sep 6 '19 at 9:09
  • $\begingroup$ Thanks @Roland and Glen_b for your comments. I am on a steep learning curve but was able to fit a much improved model using glm with a quasipoisson family. I am also looking into using weights $\endgroup$ – Fritz45 Sep 7 '19 at 0:45
1
$\begingroup$

Why does the Multiple Linear Regression model not automatically search and find the second solution if the residuals are lower?

Because the model you've implemented is not a linear model.

If you were to apply a single correction factor irrespective of the size of the predicted variable, then this is equivalent to a linear model because you are just scaling the coefficients.

But, because your correction factor is a function of the prediction, then this is no longer in the space of linear models and hence can not be estimated by linear regression.

If you need a more flexible model, I think multivariate adaptive regression splines might be a good next step for you. If I've understood your approach correctly, MARS should be a reasonable facsimile to your reweighing approach.

$\endgroup$
  • $\begingroup$ Thanks Demetri for your answer. It makes sense, my "correction factors" are defenitely not linear. I am looking at MARS regression now, I understand in R it is referred to as "earth". Thanks again. $\endgroup$ – Fritz45 Sep 6 '19 at 3:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.