Is there i.i.d. assumption on logistic regression?

Question

Is there i.i.d. assumption on the response variable of logistic regression?

For example, suppose we have $1000$ data points. It seems the response $Y_i$ is coming from a Bernoulli distribution with $p_i=\text{logit}^{-1}(\beta_0+\beta_1 x_i)$. Therefore, we should have $1000$ Bernoulli distributions, with different parameter $p$.

So, they are "independent", but are not "identical".

Am I right?

---

PS. I learned logistic regression from "machine learning" literature, where we optimize the objective function and check if it is good in testing data, without talking too much about assumptions.

My question started with this post Understand Link Function in Generalized Linear Model Where I try to learn more on statistical assumptions.

Answer 1

From your previous question you learned that GLM is described in terms of probability distribution, linear predictor $\eta$ and link function $g$ and is described as

$$ \begin{align} \eta &= X\beta \\ E(Y|X) &= \mu = g^{-1}(\eta) \end{align} $$

where $g$ is a logit link function and $Y$ is assumed to follow a Bernoulli distribution

$$ Y_i \sim \mathcal{B}(\mu_i) $$

each $Y_i$ follows Bernoulli distribution with it's own mean $\mu_i$ that is conditional on $X$. We are not assuming that each $Y_i$ comes from the same distribution, with the same mean (this would be the intercept-only model $Y_i = g^{-1}(\mu)$), but that they all have different means. We assume that $Y_i$'s are independent, i.e. we do not have to worry about things such as autocorrelation between subsequent $Y_i$ values etc.

The i.i.d. assumption is related to errors in linear regression (i.e. Gaussian GLM), where the model is

$$ y_i = \beta_0 + \beta_1 x_i + \varepsilon_i = \mu_i + \varepsilon_i $$

where $\varepsilon_i \sim \mathcal{N}(0, \sigma^2)$, so we have i.i.d. noise around $\mu_i$. This is why are interested in residuals diagnostics and pay attention to the residuals vs. fitted plot. Now, in case of GLM's like logistic regression it's not that simple since there is no additive noise term like with Gaussian model (see here, here and here). We still want residuals to be "random" around zero and we don't want to see any trends in them because they would suggest that there are some effects that are not accounted for in the model, but we don't assume that they are normal and/or i.i.d.. See also the On the importance of the i.i.d. assumption in statistical learning thread.

As a sidenote, notice that we can even drop the assumption that each $Y_i$ comes from the same kind of distribution. There are (non-GLM) models that assume that different $Y_i$'s can have different distributions with different parameters, i.e. that your data comes from a mixture of different distributions. In such case we would also assume that the $Y_i$ values are independent, since dependent values, coming from different distributions with different parameters (i.e. typical real-world data) is something that in most cases would be too complicated to model (often impossible).

Answer 2

As has been stated, while we often consider the case of iid errors in linear regression, this does not have a direct equivalent in most generalized linear models (including logistic regression). In logistic regression, we typically employ the assumption of independence of outcomes that all have a very strict relation (i.e. linear effects on the log probabilities). But these result in random variables that are not identical, nor are they decomposable into a constant term plus an iid error as is the case with linear regression.

If you really want to show that the responses have some sort of iid relation, then follow me for the next paragraph. Just know that this idea is a little off the beaten path; you may not get full credit for this response on a final if your professor is lacking in patience.

You maybe familiar with the inverse-cdf method for generating random variables. If not, here's a refresher: if $X$ has cumulative distribution function $F_X$, then I can produce random draws from $X$ by first taking random draws $q \sim \text{uniform(0,1)}$ then calculating $X = F_X^{-1}(q)$. How does this relate to logistic regression? Well, we could think that the generating process for our responses has two parts; a fixed part relating the covariates to the probabilities of success, and a random part that determines the value of the random variable conditional on the fixed part. The fixed part is defined by the link function of logistic regression, i.e. $p = \text{expit}(\beta_o + \beta_1 x)$. For the random part, let's define $F_Y( y | p)$ to be the cdf for a Bernoulli distribution with probability $p$. Then we can think of the response variable $Y_i$ being generated by the following three steps:

1.) $p_i = \text{expit}(\beta_o + \beta_1 x_i)$

2.) $q_i \sim\text{uniform(0,1)}$

3.) $Y_i = F^{-1}(q_i | p_i)$

Then the standard assumption in logistic regression is that $q_i$ is iid.