Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I am trying to do a multiple linear regression in R but am having some problems. I have a set up where I am trying to develop a multiple linear regression model for one variable (y) using six other variables ($x_{1},...,x_{6}$), all of which are correlated to some degree.

Based on my understanding of the data I ran a multiple linear regression for y using $x_3$ and $x_5$. Here is an updated residual plot based on feedback:

$$y~x_3+x5+x_3*x_5$$

IMG

How can I fix this? Would feasible GLS work? Or have I selected a bad combination of independent variables?

By the way I have limited knowledge of regression and have never done a weighted regression before so this is new to me.

Thanks for your advice!

Edit: Here are the additional plots

IMG

IMG

IMG

Here are the residual plots for $x_1...x_6$ too:

IMG

Hope this helps!

share|improve this question
    
1. What is y measuring? 2. plot y vs x3, y vs x5 and x3 vs x5. What do you see? –  Glen_b Apr 22 at 13:20
    
y is measuring benefits paid while x3 is measuring weeks claimed and x5 is measuring average weekly benefit. I should also mention there is a strong relationship between x3 (weeks claimed) and x4 (weeks compensated) and that when I run a regression with x4 and x5 I get an error due to an "essentially perfect fit". I will add a plot now. Thanks for your feedback! –  user135784 Apr 22 at 13:33
    
Ok I have updated my question with the graphs! –  user135784 Apr 22 at 13:44
1  
I see curvature and heteroskedasticity in y vs x4 and y vs x5 which suggests an issue, but it also looks like something else is going on. –  Glen_b Apr 22 at 13:54
    
Should I post plots of y against the other variables? –  user135784 Apr 22 at 13:59
show 1 more comment

4 Answers 4

up vote 3 down vote accepted

It is sometimes helpful to plot the residuals against each $x_j$ variable individually (i.e., two plots in your initial situation, possibly up to 6 plots eventually). I am not sure if that will help you here, but it's worth a try.

Looking at your top lefthand plot ("Residuals vs. Fitted"), if you had a simple (i.e., 1 $x$ variable) regression, that picture would be a tell-tale sign that the relationship between $x_j$ and $y$ is curvilinear and you need to add a squared term ($x_j^2$). In this case, you have two variables, and that implies you may have an interaction. Try adding a new variable, $x_3\times x_5$, to your model. In R, the syntax would be: y~x3+x5+x3:x5.

To illustrate my point about a missing interaction, consider this simulation:

set.seed(9)
x1  = runif(200, min=0, max=100)
x2  = runif(200, min=0, max=100)
x12 = x1*x2
y   = 0 + 1*x1 + 1*x2 + .01*x1*x2 + rnorm(200, mean=0, sd=10)
mod = lm(y~x1+x2)

There is an interaction in the data generating process, but the model is mis-specified, it leaves the interaction out. Here is what the plots look like:

enter image description here

Note that a missing curvilinear term (x^2) could cause a similar picture. There is a little squirrelly curvature between y and x5, so a squared term might help there a little bit.

share|improve this answer
    
Ok thanks for your help Gung! I will try this tomorrow and see how it works out as I am very tired right now. –  user135784 Apr 22 at 14:31
    
I have updated the residual plots with the model you suggested. They have improved but are not completely fixed yet. What do you recommend? Thanks! –  user135784 Apr 23 at 10:21
    
Hmmm, can you post a csv file of your data, say to dropbox or similar? –  gung Apr 23 at 15:55
add comment

First of all get read of the outlier with too big cook distance and do the regression again.

Try to add x² in the regressor when y vs x does not look linear.

If the variance of your residuals is still increasing with y after that, perhaps an heteroscedastic regression would be better.

If it still does not work you can try a non-parametric regression.

share|improve this answer
2  
I think these ideas would often be reasonable, but they may not be right in this case. The 'outliers' identified in the Cook's distance plot are contiguous w/ the rest of the data; the model is more likely misspecified than it is that those data come from a different distribution. Adding $x^2$ would be indicated if the res v fitd plot were of a single variable, but may not be w/ 2 vars. –  gung Apr 22 at 14:26
    
Thanks Mothas! I might give this a try in addition to Gung's tips –  user135784 Apr 22 at 14:34
add comment

If you don't want to abandon the lm setting because you're not familiar with regressions, my solution could help.

I've noticed you use R, so what I show you may be useful:

I suggest you car::residualPlots function to have a clearer idea of how the sample residuals behave against each predictor and against fitted values.
This function provides both the plots with the curve pattern and the tests for curvature of residuals. In your case, (some or all) tests will reject the NH of no curvature.
This does not imply that adding a quadratic term for each predictor will surely solve your problem, instead you'd rather think this tests as a proof that something is not fine. Then you can:

1) transform predictors (in this case car::powerTransform may provide useful hints);

2) add quadratic, cubic,... or interaction terms;

3) if something still looks bad, transform also the response (you should put your model formula inside car::powerTransform).

Then do the diagnostics (outliers and cook's distance, heteroschedasticity..);

if everything satisfies you, run the summary.

share|improve this answer
    
Thanks Davide that explanation helped out I will try out those functions and see how it turns out! –  user135784 Apr 23 at 0:08
add comment

I'm not allowed to comment yet, but if I understood the variables correctly, then you are expecting an interaction term, y ~ x3 * x5, since benefits paid should be equal to avg benefit per week times number of weeks (or at least more equal to that than any linear combination).

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.