I am trying to make some sense out of the results of a linear regression model. I have a dependent variable X, and, say, 3 independent variables Y1 Y2 Y3. I set up 5 models :

(m1) X ~ Y1

(m2) X ~ Y1 + Y2

(m3) X ~ Y1 + Y3

(m4) X ~ Y1 + Y2 + Y3

(m5) X ~ Y1 + Y2*Y3

In models 1 to 3, all independent variables are significant.

In model 4, when I include both Y2 and Y3 in the model, they both turn un-significant. However, when I include an interaction between Y2 and Y3 (m5), all variables main effects and the interaction turn significant.

I am wondering whether I am overfitting the model, or if there might be one logical way to interpret the changes of significances for Y2 and Y3 in these analyses (m4 vs m5).

With regards

[edit] Y2 is numeric, Y3 is factor (3 levels). The parameters' sign are the following : (m5) main Y2 : +, main Y3 : +, Y2*Y3[2] : -, Y2*Y3[3] : -,

  • $\begingroup$ Which sign do the resulting parameters have for each variable? Ara all of them numeric variables or are there factors? $\endgroup$
    – Rufo
    Jun 5 '14 at 10:09
  • $\begingroup$ What is the size of your sample? What is the goal of your model? (inference or prediction in future samples) $\endgroup$ Jun 5 '14 at 12:16
  • $\begingroup$ N=44, not perfectly balanced accross factor levels. The goal is not predictions at this stage, just finding the best way to model sample data variance. $\endgroup$
    – tacotac
    Jun 6 '14 at 7:30

@Vincent's approach works quite well, but I prefer using a likelihood ratio test to compare nested models such as this.

For example, we could use the below R code to compare model 4 and model 5:

 m4<-lm(X ~ Y1 + Y2 + Y3)
 m5<-lm(X ~ Y1 + Y2*Y3)

If the results of this likelihood ratio test are non-significant, using model 5 provides no additional information than model 4, and the simpler model (model 4) should be used instead of the more complex model (model 5).

If the results of this likelihood ratio test are significant, using model 5 provides more information than model 4, and the more complex model (model 5) should be used instead of the simpler model (model 4).

Additionally, you can address the issue of overfitting by assessing the adjusted R-squared values of each of these models by using the below R code's output:


Although m5's "Multiple R-squared" (or simply, R-squared) will more than likely be higher than the R-squared for m4, the "Adjusted R-Squared" may or may not be. If the Adjusted R-Squared is lower for m5, then m5 is overfitting the model and m4 should be used, whereas if the Adjusted R-Squared is higher for m5, then m5 should be used as it is providing additional information without overfitting our model to the sample data.

Lastly, to further address the issue of overfitting, I would highly recommend using a hold out sample (splitting the full data into training and validation datasets - if sample size permits), using the training dataset to build the model to predict the outcome in the validation dataset. The model with the lowest validation set Root Mean Squared Error (RMSE), is the most preferred model. I hope this helps!

  • $\begingroup$ The leave-k-out validation would have been a great option but my dataset size do not allow such luxury... $\endgroup$
    – tacotac
    Jun 5 '14 at 12:06

You can find each model's AIC/BIC, (or other information criteria) and find the best one.

Or do a Chi-square test, comparing two nested models step by step.

Individual t test is not telling us which variable should be included or removed.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.