# In my logistic regression model one of the independent variables is redundant with the interaction term. How should I deal with it?

In my logistic regression is the dependent variable a dummy variable and I also have two independent variables. One of those is a dummy variable and the other is a metric variable. I also suppose an interaction between those two variables.

I am computing three regressions because I want to explore the influence of the independent variables on the dependent variable in period 1, period 2 and in both periods together.

When I compute the regressions for period 2 and for both periods together, there is no problem.

But when I compute the regression for the period 1, SPSS generated a warning that "because of redundancy the degrees of freedom for at least one variable have been reduced". I actually do not know what that means, but I found out that when I exclude the independent dummy variable from my model for period 1, the interaction term is included in the model. So both variables are somehow identical.

My question is how should I deal with this in my work. Should I just say that for the first period the interaction term and the dummy variable are identical? Or there are some other consequences for the interpretation of my model?

I hope this makes my question a little bit clearer. Thanks again.

• Welcome to the site. You can read the FAQ here. What do you mean exactly by "redondant"? If your independent variable and your interaction term always have the same value, you can't put the two of them in a model, because they provide exactly the same information, so any modern statistical package is going to exclude one of the variable from the estimation. But maybe I am misunderstanding what you mean by redondant, in which case you will need to develop your question a bit to make it possible to answer it. Commented Mar 20, 2013 at 17:33

Software will drop variables when they are collinear. Understanding this situation amounts to figuring more precisely what that means.

There are three independent variables involved, including the constant term. Let's represent their values as the constant (column) vector $X_1 = (1, 1, \ldots, 1)$, a vector of ones and zeros for the dummy $X_2 = (1, 1, \ldots, 1, 0, 0,\ldots, 0)$, and a third apparently arbitrary vector $X_3 = (x_1, x_2, \ldots, x_n)$. (All other valid dummy codings are linear combinations of this particular $X_1$ and $X_2$, so no generality is lost by assuming that this particular binary (0-1) encoding is used.) I have sorted the data so that all the records where the dummy is $1$ come first; suppose there are $k$ of them. (We know $k \ge 1$ and $k \lt n$, for otherwise the dummy would be constant and could not be included in any regression with a constant term.)

Collinearity of these three vectors along with the $X_2 X_3$ interaction means (by definition) that there is a nontrivial linear relation

$$0 = \alpha_1 X_1 + \alpha_2 X_2 + \alpha_3 X_3 + \alpha_4 X_2 X_3$$

The first $k$ equations in this linear combination are

$$0 = \alpha_1 + \alpha_2 + \alpha_3 x_i + \alpha_4 x_i,\quad i=1, 2, \ldots, k.$$

The remaining equations are

$$0 = \alpha_1 + \alpha_3 x_i,\quad i = k+1, \ldots, n.$$

The first group of equations informs us that all the $(\alpha_3 + \alpha_4)x_i$ are equal to the constant $-(\alpha_1+\alpha_2)$ for $1 \le i \le k$. The second group informs us that all the $\alpha_3 x_i$ are equal to the constant $-\alpha_1$ for $k \lt i \le n$. That first statement does not restrict the $x_i$ for $1 \le i \le k$ provided $\alpha_3 + \alpha_4=0$, but the second one then implies that all the $x_i$ are equal to one another for $i \gt k$. For if this were not the case, then necessarily $\alpha_3 = 0$, implying either $\alpha_4=0$ or all the $x_i$ are equal to each other for $1\le i \le k$. If $\alpha_4=0$, these would in turn imply that both $\alpha_1 + \alpha_2=0$ and $\alpha_1=0$, reducing all the $\alpha_i$ to $0$: but that was not the case (the linear relation was nontrivial).

In words, what we have deduced is that the continuous variable $X_3$ exhibits no variation among at least one of the two groups of dummy values.

To confirm this conclusion we may create three examples of such data in R. I have chosen $k=2$ and $n=4$: there are two records for each group of dummy values. In the first case, assigning random values to $X_3$ virtually guarantees there will be variation within both groups:

> set.seed(17)
> x2 <- c(1, 1, 0, 0) # The dummy (binary) variable, sorted as in the analysis
> x3 <- rnorm(4)      # The continuous independent variable
> y <- rnorm(4)       # The dependent variable may have *any* values
> lm(y ~ x2*x3)
Coefficients:
(Intercept)           x2           x3        x2:x3
0.6763      -0.9218      -1.2728       0.2703


All variables are retained. (This is OLS regression, not logistic regression, but that doesn't matter: both methods behave identically concerning treatment of collinear independent variables.)

In the second case, let's set the first two first elements of $X_3$ to the same value:

> x3[1] <- x3[2]; lm(y ~ x2*x3)
Coefficients:
(Intercept)           x2           x3        x2:x3
0.6763      -0.4745      -1.2728           NA


The interaction is dropped due to the collinearity.

In the third case, let's set the last two elements of $X_3$ to a common value while varying the first two. To do this, I just reverse all the element of $X_3$:

> x3 <- rev(x3); lm(y ~ x2*x3)
Coefficients:
(Intercept)           x2           x3        x2:x3
1.217       -1.756       -1.605           NA


Once again the interaction is dropped due to collinearity.

It sounds like SPSS behaves in the same way as R in such cases.

• Thank you so much for your detailed explanations.This is really a great explanation! You helped me a lot. Commented Mar 21, 2013 at 11:04

Sounds like your independent variables are correlated. Likely it is the case that when one is 0, the other is 1, or one of the variables takes only one value. I am not familiar with SPSS, but it could also be an error in your code, so I would advise you to post it up for others to help.

Also, it shouldn't surprise you that the model changes when you remove a term. That does not, however, mean that the interaction term is identical to the removed term. This would only be the case where all of one variable is zero, all of one variable is one, or they are inverse of each other. This is because the interaction var1*var2 is simply var1 times var2 with the only possible values being 1 and 0. The only way your interaction for a record is not zero is if both the other variables are 1.

• No problem. I am curious if you were able to solve the problem after the answers given
– TLJ
Commented Mar 21, 2013 at 17:45