1
$\begingroup$

I noticed that I can substantially improve my model's R$^2$ and Residual standard error values by adding some interaction terms.

My model's statistics go from:

Residual standard error: 52.06 on 790 degrees of freedom
Multiple R-squared:  0.5862,    Adjusted R-squared:  0.4773 
F-statistic: 5.381 on 208 and 790 DF,  p-value: < 0.00000000000000022

to

Residual standard error: 45.22 on 106 degrees of freedom
Multiple R-squared:  0.9581,    Adjusted R-squared:  0.6057 
F-statistic: 2.718 on 892 and 106 DF,  p-value: 0.0000000009271

but look what happens to the plots:

from:

enter image description here

to:

enter image description here

tells me that adding the interaction terms is causing something very bad to the model. Or could this signify that actually my data set is e.g. highly non-linear, which is why I would get a "better fit" (according to the summaries above), eventhough the plots start to show more non-linear behaviour?

The reason for believing in "non-linearity" is that:

In linear models I've heard there to be an assumption of the errors having to be normally distributed, that would mean that the QQ-plot should be as close to the straight line as possible. Right? So my thinking of "non-linearity" came from thinking that "linearity assumptions" are violated.

Also, does violating the linearity assumptions reflect to the predictions made on "new data", but not on the data (observations) that's used for the fit? So the reason to take linearity assumptions seriously would be so that the model wouldn't be dependent on the original data set and can work in predicting "future observations"?

How should I balance and interpret these results and what model should I continue to develop, the first one or the second one?

$\endgroup$
3
  • $\begingroup$ Just to better understand your situation and put Glen_b's answer in context: how many predictors and observations do you have? If you have about 800 observations and 25 predictors, by including interactions you may be just overfitting your model. $\endgroup$
    – Pere
    Oct 16, 2016 at 12:29
  • $\begingroup$ @Pere I have 1000 observations and 10 variables in total of which the best current model (the latter plots above) looks like: lm(dta$X.U.FEFF..mpist. ~ factor(dta$id)*factor(dta$sukup) + factor(dta$aidink)*factor(dta$id) + factor(dta$aidink)*factor(dta$sukup) + factor(dta$matem)*factor(dta$sukup) + factor(dta$matem)*factor(dta$HISEI) + factor(dta$aidink)*factor(dta$HISEI)). So this model uses only 5 out of 10 variables, since I found the remaining 5 to not be significant. $\endgroup$
    – mavavilj
    Oct 16, 2016 at 12:32
  • 2
    $\begingroup$ If the five or some if the five variables are categorical, then note that each of these categorical variables is in fact 'replaced' by a number of dummies. $\endgroup$
    – user83346
    Oct 16, 2016 at 13:15

1 Answer 1

3
$\begingroup$

A non-straight Q-Q plot indicates non-normal residuals, not "nonlinearity in the model". That may be due to a variety of model issues. In this case the reason for the flat sections in the Q-Q plot seem clear enough.

The hundreds of interaction terms in the model (684 df worth) result in you getting exact or nearly exact fits to a substantial fraction of your data. The resulting zero residuals lead to the "line" across the residuals vs fitted and to the large flat part in the normal Q-Q plot.

This could occur (for example) when you have many factor combinations corresponding to a single observation (which would then be fitted exactly). Smaller flat sections might correspond to factor combinations with two observations (or even 3 or 4 if the response is discrete).

It doesn't necessarily suggest anything is nonlinear; it might be purely to the enormous number of interaction terms.

If you want to add all of those terms I'd suggest considering some form of regularization.

$\endgroup$
5
  • $\begingroup$ So do you suggest going on with the interaction terms, even though the QQ-plot seems to lose its interpretation, since I cannot look for a "straight line" any more? What explains the curvature in QQ-plot? Can I trust that the summary() statistics getting better means that the model is getting better, eventhought the plots may start to look strange? Where to draw the line between looking at summary() and plots? $\endgroup$
    – mavavilj
    Oct 16, 2016 at 11:36
  • $\begingroup$ In linear models I've heard there to be an assumption or the errors having to be normally distributed, that would mean that the QQ-plot should be as close to the straight line as possible. Right? So my thinking of "non-linearity" came from thinking that "linearity assumptions" are violated. $\endgroup$
    – mavavilj
    Oct 16, 2016 at 11:37
  • $\begingroup$ @mdewey -- indeed, that's literally what a fair fraction of my answer is discussing. $\endgroup$
    – Glen_b
    Oct 16, 2016 at 21:01
  • 1
    $\begingroup$ @mavavilj I don't have nearly enough context to suggest you include or exclude anything nor enough to suggest much in the way of alternatives (but I would suggest you consider the folk theorem of statistical computing). I would not normally include so many interaction terms -- many seemingly with only one observation for an effect -- without some form of regularization. You might consider mixed effects models perhaps but there's nothing to go on (you may need to post a new question). $\endgroup$
    – Glen_b
    Oct 16, 2016 at 22:43
  • 1
    $\begingroup$ @mavavilj In relation to your second comment -- as my post already explains the likely cause of the flat parts of your Q-Q plot (nearly 700 interaction terms with not all that many more points), I don't think we have any clear basis to think it could be nonlinearity. $\endgroup$
    – Glen_b
    Oct 16, 2016 at 22:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.