Significance of independent variables in linear regression models

Question

I am trying to make some sense out of the results of a linear regression model. I have a dependent variable X, and, say, 3 independent variables Y1 Y2 Y3. I set up 5 models :

(m1) X ~ Y1

(m2) X ~ Y1 + Y2

(m3) X ~ Y1 + Y3

(m4) X ~ Y1 + Y2 + Y3

(m5) X ~ Y1 + Y2*Y3

In models 1 to 3, all independent variables are significant.

In model 4, when I include both Y2 and Y3 in the model, they both turn un-significant. However, when I include an interaction between Y2 and Y3 (m5), all variables main effects and the interaction turn significant.

I am wondering whether I am overfitting the model, or if there might be one logical way to interpret the changes of significances for Y2 and Y3 in these analyses (m4 vs m5).

With regards

[edit] Y2 is numeric, Y3 is factor (3 levels). The parameters' sign are the following : (m5) main Y2 : +, main Y3 : +, Y2*Y3[2] : -, Y2*Y3[3] : -,

Answer 1

@Vincent's approach works quite well, but I prefer using a likelihood ratio test to compare nested models such as this.

For example, we could use the below R code to compare model 4 and model 5:

 attach(your_dataset)
 m4<-lm(X ~ Y1 + Y2 + Y3)
 m5<-lm(X ~ Y1 + Y2*Y3)
 anova(m4,m5)

If the results of this likelihood ratio test are non-significant, using model 5 provides no additional information than model 4, and the simpler model (model 4) should be used instead of the more complex model (model 5).

If the results of this likelihood ratio test are significant, using model 5 provides more information than model 4, and the more complex model (model 5) should be used instead of the simpler model (model 4).

Additionally, you can address the issue of overfitting by assessing the adjusted R-squared values of each of these models by using the below R code's output:

 summary(m4)
 summary(m5)

Although m5's "Multiple R-squared" (or simply, R-squared) will more than likely be higher than the R-squared for m4, the "Adjusted R-Squared" may or may not be. If the Adjusted R-Squared is lower for m5, then m5 is overfitting the model and m4 should be used, whereas if the Adjusted R-Squared is higher for m5, then m5 should be used as it is providing additional information without overfitting our model to the sample data.

Lastly, to further address the issue of overfitting, I would highly recommend using a hold out sample (splitting the full data into training and validation datasets - if sample size permits), using the training dataset to build the model to predict the outcome in the validation dataset. The model with the lowest validation set Root Mean Squared Error (RMSE), is the most preferred model. I hope this helps!

Answer 2

You can find each model's AIC/BIC, (or other information criteria) and find the best one.

Or do a Chi-square test, comparing two nested models step by step.

Individual t test is not telling us which variable should be included or removed.