Significance of Categorical variables in regression in R - against each other

Question

Good afternoon, I need help in intpretting the significance results when performing a linear regression with a categorical interaction. I'm running this analysis in R.

The question is whether I can use qsec as an explantory variable in the below model given that one of the categorical variables is not significantly different to the other.

Model & Outputs

lm(mpg~qsec+am+qsec*am, data=mtcars)

Coefficients:
            Estimate Std. Error t value Pr(>|t|)   
(Intercept)  -9.0099     8.2179  -1.096  0.28226   
qsec          1.4385     0.4500   3.197  0.00343 **
am1         -14.5107    12.4812  -1.163  0.25481   
qsec:am1      1.3214     0.7017   1.883  0.07012 . 

The second coefficient is not statistical significant - though only just with a p value of 0.07. Does this mean that I should look for another explanatory variable?

When I run the regression on one of the values - am1 - alone, then it is still significantly different to zero. As is am0 from the above results.

When I look at the data, the slope appears significantly different so I think I can use it however the understanding of the theory says no - which is frustrating. I would add my plot to highlight my point but I'm struggling to load it.

I hope this isn't a stupid question to ask...I've found some good explanations of categorical variables in general, however nothing that tells me how I should interpret this secondary variable.

James

** edit changed data to be qsec per question below

Answer 1

This is an area where exploring the data and your results should help guide your conclusions. Remember that "statistical significance" is not the end-all of a conclusion. A good analysis will help the data tell it's own story, and relying on significance is only looking at part of the story. First, let's plot our data:

library(dplyr)
library(ggplot2)
library(tidyr)

mtcars$am <- factor(mtcars$am, 0:1, c("Automatic", "Manual"))

ggplot(data = mtcars,
       aes(x = qsec,
           y = mpg,
           colour = am)) + 
  geom_point()

You should seem some pretty clear trends in these data (trends this clear are kind of rare). The Manual transmissions tend to have higher mpg, and in general, higher qsec (quarter mile time) tracks with higher mpg.

When we fit the "plain" model, our results give objective evidence to those suspicions:

#* Fit the model without the interaction
fit0 <- lm(mpg ~ qsec + am, data = mtcars)
summary(fit0)

As you've noticed, that evidence isn't as simple to see once we include the interaction term:

#* Fit the model with the interaction
fit1 <- lm(mpg ~ qsec * am, data = mtcars)
summary(fit1)

So what happened? The interaction term is "modifying" the effect of qsec on mpg differently for manual transmissions than it is for automatic transmissions. In other words, the plain model assumes that the slopes are identical in both groups, while the interaction model accepts the possibility that the slopes are different. This can be visualized in the following figure:

new_data <- expand.grid(qsec = seq(min(mtcars$qsec), 
                                       max(mtcars$qsec), 
                                   length.out = 1000),
                        am = c("Automatic", "Manual")) %>%
  mutate(pred_plain = predict(fit0, newdata = .),
         pred_interaction = predict(fit1, newdata = .)) %>%
  gather(model, pred,
         pred_plain, pred_interaction) %>%
  mutate(model = gsub("pred_", "", model))

ggplot(new_data,
       aes(x = qsec,
           y = pred,
           colour = am)) + 
  geom_line() + 
  ylab("Predicted mpg") + 
  xlab("Quarter Mile Time (qsec)") + 
  facet_grid(model ~ .)

From the figure, we see a pretty clear indication that the increase in mpg as qsec increases is greater in the manual transmission than the automatic transmission.

One of the more difficult questions to answer, based on our results so far, is whether or not the transmission as a significant effect in this model. Recall from the summary of fit1 that the coefficient for am is not statistically significant, but it was in fit0, and the predicted values suggest that maybe it has a meaningful impact. In these situations, it's important to remember that you shouldn't interpret a main effect without its accompanying interaction effects. How to interpret them together is a challenging topic, and I'm not sure I'm well qualified to address it. But I will point out that the model summary may not be the best place to look as it focuses on the coefficients, not necessarily the variables. Instead, take a look at the ANOVA summary

anova(fit1)
Analysis of Variance Table

Response: mpg
          Df Sum Sq Mean Sq F value    Pr(>F)    
qsec       1 197.39  197.39 17.6583 0.0002438 ***
am         1 576.02  576.02 51.5297 8.203e-08 ***
qsec:am    1  39.64   39.64  3.5458 0.0701227 .  
Residuals 28 313.00   11.18                      
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

This summary gives a better overview of how "impactful" each variable is, as opposed to just the regression coefficients. It suggests that qsec and am are both related to the change in response, even in the presence of the interaction.

The other difficult question to answer is if the interaction is worth keeping in the model. This is like asking if the interaction modified the slopes enough to make us believe that they really aren't parallel. I believe there are a number of ways to make this decision, and honestly, ask me on a different day and I may give you a different answer. Some people say to remove interactions if they are not statistically significant, but today I say to keep them if they show a p-value less than 0.10 because I'd prefer to err on the side of a slightly more complex model than to over simplify the model. But the decision may involve as much experience and subject matter knowledge as it does objective science.