3
$\begingroup$

I have a categorical variable that has $4$ categories, and I have two dummy variables, $x_1$ and $x_2$, that cover this categorical variable. The $x_1$ variable has values of only $1$ without any zeroes $[1,1,1,1,1...1]$, while $x_2$ has values of both $0$ and $1$ $[0,1,1,0,0,0...0]$. When I use multiple linear regression model with these two dummy variables as regressors, I get missing values at the output of summary(model) function in R (NA).

My questions are:

  1. Should I keep the regressor or remove it from the model?
  2. What will happen to the interpretations of the parameter estimates when I remove the intercept?
  3. Is there a situation where can we freely remove the intercept without affecting the interpretation of the parameter estimates?
  4. Why does R-squared increase when I remove the intercept?

This is the model with the intercept:

Call:
lm(formula = output ~ X1 + X2, data = data)

Residuals:
    Min      1Q  Median      3Q     Max 
-3.3091 -1.2591 -0.0091  1.2415  3.6909 

Coefficients: (1 not defined because of singularities)
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)  10.3083     0.5329  19.345   <2e-16 ***
X1                NA         NA      NA       NA    
X2           -0.3992     0.6012  -0.664    0.509    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 1.846 on 54 degrees of freedom
Multiple R-squared:  0.008101,  Adjusted R-squared:  -0.01027 
F-statistic: 0.441 on 1 and 54 DF,  p-value: 0.5094

This is the model without the intercept:

Call:
lm(formula = output ~ X1 + X2 - 1, data = data)

Residuals:
    Min      1Q  Median      3Q     Max 
-3.3091 -1.2591 -0.0091  1.2415  3.6909 

Coefficients:
   Estimate Std. Error t value Pr(>|t|)    
X1  10.3083     0.5329  19.345   <2e-16 ***
X2  -0.3992     0.6012  -0.664    0.509    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 1.846 on 54 degrees of freedom
Multiple R-squared:  0.9682,    Adjusted R-squared:  0.967 
F-statistic: 821.1 on 2 and 54 DF,  p-value: < 2.2e-16

This is the model without the regressor X1:

Call:
lm(formula = output ~ X2, data = data)

Residuals:
    Min      1Q  Median      3Q     Max 
-3.3091 -1.2591 -0.0091  1.2415  3.6909 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)  10.3083     0.5329  19.345   <2e-16 ***
X2           -0.3992     0.6012  -0.664    0.509    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 1.846 on 54 degrees of freedom
Multiple R-squared:  0.008101,  Adjusted R-squared:  -0.01027 
F-statistic: 0.441 on 1 and 54 DF,  p-value: 0.5094
$\endgroup$

1 Answer 1

2
$\begingroup$

I think the issue is quite obvious from what I can see. Your $x_1$ variable here only takes on one value, therefore it is simply a linear constant aka it doesn't contribute any new information to the model (it also appears there are problems with $x_2$ but I'll remark on that later). To your questions:

Should I keep the regressor or remove it from the model?

Remove it. It adds nothing to your model.

What will happen to the interpretations of the parameter estimates when I remove the intercept?

With respect to a regression that only includes categorical predictors, there is nothing wrong with removing the intercept. It simply makes the intercept/slopes more interpretable (they just represent the contrasts between levels of each factor).

Is there a situation where can we freely remove the intercept without affecting the interpretation of the parameter estimates?

In this case it isn't an issue. It can also be okay to do when $x=0$ and $y=0$ is a real possibility (Eisenhauer, 2003; Hahn, 1977), but generally speaking it is better to leave the intercept in.

Why does R-squared increase when I remove the intercept?

This is normal. Suppressing the intercept means that you are just predicting the model based off mean adjustments between groups, which without including any slope terms, means that you are capturing most/all of the variation in the DV as apportioned by each group. But again I think that $x_1$ here doesn't add anything to your model so you can simply remove it. Notice that Model 1 and 3 have no change despite actively removing the variable.

One final note...your $x_2$ variable doesn't seem to contribute much either though. This is why your adjusted $R^2$ is also negative (see a visualization of this issue here with some brief explanation in Chicco et al., 2021). Your model basically has no predictive value. As an example of a dummy coded (binary) variable fit with no relationship to the outcome:

#### Simualate Data ####
set.seed(123)
n <- 200
x <- rbinom(n,1,.5)
y <- rnorm(n)

#### Fit and Summarize ####
fit <- lm(y ~ x+0)
summary(fit)

#### Plot ####
plot(x,y,
     main="Negative R2 Fit")
abline(fit,
       col="red",
       lwd=2)

We get a negative $R^2$, and our regression line fit is close to horizontal (note that using an intercept or suppressing it here doesn't make a difference, I just show a similar case here to yours).

enter image description here

Edit

It occurs to me now that you also haven't coded your variable as a factor. The issue is it's not coding your 0 and 1 values as actual dummy codes, though I'm not sure if that actually fixes your problem (for example y ~ factor(x)). Suppressing categorical regression intercepts doesn't affect the fitting and only changes the interpretation (they simply represent the contrast in means between groups). For an example of what I mean, here I fit a similar set of simulated data with 4 levels this time and coded as numeric and as factor:

#### Set Seed ####
set.seed(123)
n <- 200
x2 <- rbinom(n,3,.5)
y2 <- rnorm(n)

#### Fit and Summarize ####
fit.num <- lm(y2 ~ x2 - 1)
fit.cat <- lm(y2 ~ factor(x2)-1)
summary(fit.num)
summary(fit.cat)

You can see one estimates the slope as a numeric variable, whereas the other actually contrasts by group (dummy codes). Both are still poorly fit.

References

$\endgroup$
6
  • $\begingroup$ in this example, the categorical variable has 4 values, 1, 2, 3, and 4. It does not have the value of 0, so, can I safely remove the intercept in this example, without affecting the interpretation of the parameter estimates? $\endgroup$
    – user400487
    Nov 12, 2023 at 12:12
  • $\begingroup$ See my edit to the answer. $\endgroup$ Nov 12, 2023 at 12:30
  • $\begingroup$ Thank you for your answer, I did not do the factor(x), the question is from an excercise, and it asks whether the intercept should be included or exluded from the additive model, and how will removing or keeping it affect the interpretations of the parameter estimates. $\endgroup$
    – user400487
    Nov 12, 2023 at 12:39
  • $\begingroup$ If this is for homework, I would advise testing out the suppression of intercepts with real data too to see when it does and doesn't work. But the articles I referenced at the end of my answer explain cases when it is useful and not so useful. $\endgroup$ Nov 12, 2023 at 12:43
  • $\begingroup$ Thank you,If I have 2 regressors in regression, with low vif (less than 5 for each of them), after squaring them, the vif increased significantly (more than 10), in the course, we haven’t covered how to fix high vif. So, is it possible to interprete the parameter estimates of these regressors, while having very high vif due to the squaring of the regressors? $\endgroup$
    – user400487
    Nov 12, 2023 at 14:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.