# Effect of zero-variance variables on logistic regression

Imagine that due to pipeline/workflow issues a logistic regression model that's in production has some binary variables in it that are positive or negative for all observations in the modeled population.

Is there any way for these "constant" variables to affect the model, and what exactly is happening mathematically either way? Do the coefficients simply come out as zero and the variables mathematically drop out of the regression, so that you could add a thousand variables like this and the regression wouldn't be affected? Would this work the same way for linear regression as it would for logistic?

Edit: Note that there are some normal varying variables in the model, so these non-varying/constant variables are in addition to those.

This is known as "perfect collinearity". "The model" doesn't exist, but typically software implementations have a few tricks to return a model when this happens.

Generally, suppose that $$X$$ is my design matrix and that I can write a single column of $$X$$ as a linear combination of the others, so

$$x_j = \sum_{i \neq j} c_i x_i$$

We would say that $$X$$ then displays perfect collinearity between predictors. Remember that one column of $$X$$ is just a column of ones. This is for the intercept of the model, so let's denote it $$\mathbf{1}$$. In the case where $$x_j$$ is all 0s then $$x_j = c_1 \mathbf{1}$$, where $$c_1 = 0$$. The case where $$x_j$$ is a column of a single non-zero number is a straight forward generalization.

The consequence of fitting a linear model to a design matrix which has perfectly collinear columns is that the OLS estimates are underdetermined; they are not unique. This is fairly simple to show. Suppose $$y = \beta_0 + \beta_1 x + \beta_2 x$$ is a model we want to fit (for simplicity here I have duplicated $$x$$ in the model on purpose meaning we have perfect collinearity with $$x$$ itself). Well, clearly, I can rewrite the model as

$$y = \beta_0 + (\beta_1 + \beta_2)x$$

meaning if you give me a $$\beta_1$$ I can give you back a $$\beta_2$$ so that the model fits the data as best it can. There are hence an infinite number of coefficients consistent with the data, and the solution is thus not unique.

Typically we say we can't perform the regression in this case. With good reason, the OLS estimates are $$\beta = (X^T X)^{-1}X^T y$$. If $$X$$ is rank deficient, then so is the Grahm matrix $$X^TX$$ and so the inverse does not exist. But lots of software implementations of linear models have catches for this sort of thing and usually just 0 out perfectly collinear columns. R makes the variable NA rather than 0

x = rnorm(100)
z = rnorm(100)
w = x+z # w is collinear with x and z
y = 2*x + z - w + rnorm(100)
lm(y~x+z+w)
Call:
lm(formula = y ~ x + z + w)

Coefficients:
(Intercept)            x            z            w
0.08462      1.08743      0.16874           NA


Sklearn does something different depending on if the predictor has 0 variance or not. You're free to explore documentation of popular libraries how each handles this.

Logistic regression has a similar story, though the mathematics will be a bit more difficult. For simplicity, I will rely on the intuition we have built here. Logistic regression is fit my maximizing the negative log-likelihood function. When a variable is collinear with others, then the optima is not unique; there is an infinite number of parameters which maximize the likelihood. Again, different software handles this differently, and I would encourage you to read documentation to be clear on how collinearity is handled in each case.

• Thanks! Does all of this apply equally / in the same way if there are some normal variables in the regression in addition to some variables with only one unique value? If the software doesn't have some special handling for these 0 variance variables, what will happen? Commented Feb 28, 2022 at 20:33
• @semblable Yes, if there are variables with non-zero variance the problem persists. If the software doesn't have explicit handling for this type of problem, then an error will likely occur. OLS requires solving a linear system, which is degenerate when vars are colinear. GLM will likely fail to converge as the optima is non-unique. Commented Feb 28, 2022 at 20:40