18
$\begingroup$

Is there i.i.d. assumption on the response variable of logistic regression?

For example, suppose we have $1000$ data points. It seems the response $Y_i$ is coming from a Bernoulli distribution with $p_i=\text{logit}(\beta_0+\beta_1 x_i)$. Therefore, we should have $1000$ Bernoulli distributions, with different parameter $p$.

So, they are "independent", but are not "identical".

Am I right?


PS. I learned logistic regression from "machine learning" literature, where we optimize the objective function and check if it is good in testing data, without talking too much about assumptions.

My question started with this post Understand Link Function in Generalized Linear Model Where I try to learn more on statistical assumptions.

$\endgroup$
  • 1
    $\begingroup$ An "assumption" is something that a theorem can have. Linear regression has an "assumption" of iid errors (it's not $y$s that are "assumed" to be iid in linear regression! it's the errors) in the sense that the Gauss-Markov theorem has this assumption. Now, is there any theorem that one has a mind for logistic regression? If not, then there are no "assumptions". $\endgroup$ – amoeba Feb 3 '17 at 0:04
  • 7
    $\begingroup$ @Amoeba, hxd is correct in noting the distributions are not identical: "iid" does not apply. If one is using logistic regression only for its fit, then (as you write) perhaps few assumptions are needed; but as soon as one makes use of the estimated covariance matrix of the coefficients or wishes to construct prediction intervals (or, for that matter, cross-validate predicted values), then that requires probabilistic assumptions. The usual one is that the responses are independent. $\endgroup$ – whuber Feb 3 '17 at 0:09
  • 4
    $\begingroup$ @amoeba once you want perform inference (hypothesis tests, confidence intervals etc) rather than simply calculate estimates of parameters, you will make a slew of assumptions (some more critical than others) in order to be able to derive the relevant null distribution of the test statistic or the necessary calculations for an interval with the desired coverage. Even relatively low-assumption procedures still have assumptions, and if we care about our inferences, we will care about whether they are likely to have something near their nominal properties. $\endgroup$ – Glen_b Feb 3 '17 at 1:31
  • 1
    $\begingroup$ @amoeba, I like a theorem that shows the asymptotic normality of the MLE. I also like the likelihood ratio test. $\endgroup$ – gammer Feb 3 '17 at 1:35
  • 2
    $\begingroup$ Their marginal distributions are not identical unless they all have the same predictor value, in which case you just have IID bernoulli trials. Their conditional distributions (given the predictor) are all the same, but I don't think you'd normally say the $Y_i$ in this case are IID. $\endgroup$ – gammer Feb 3 '17 at 1:38
10
$\begingroup$

From your previous question you learned that GLM is described in terms of probability distribution, linear predictor $\eta$ and link function $g$ and is described as

$$ \begin{align} \eta &= X\beta \\ E(Y|X) &= \mu = g^{-1}(\eta) \end{align} $$

where $g$ is a logit link function and $Y$ is assumed to follow a Bernoulli distribution

$$ Y_i \sim \mathcal{B}(\mu_i) $$

each $Y_i$ follows Bernoulli distribution with it's own mean $\mu_i$ that is conditional on $X$. We are not assuming that each $Y_i$ comes from the same distribution, with the same mean (this would be the intercept-only model $Y_i = g^{-1}(\mu)$), but that they all have different means. We assume that $Y_i$'s are independent, i.e. we do not have to worry about things such as autocorrelation between subsequent $Y_i$ values etc.

The i.i.d. assumption is related to errors in linear regression (i.e. Gaussian GLM), where the model is

$$ y_i = \beta_0 + \beta_1 x_i + \varepsilon_i = \mu_i + \varepsilon_i $$

where $\varepsilon_i \sim \mathcal{N}(0, \sigma^2)$, so we have i.i.d. noise around $\mu_i$. This is why are interested in residuals diagnostics and pay attention to the residuals vs. fitted plot. Now, in case of GLM's like logistic regression it's not that simple since there is no additive noise term like with Gaussian model (see here, here and here). We still want residuals to be "random" around zero and we don't want to see any trends in them because they would suggest that there are some effects that are not accounted for in the model, but we don't assume that they are normal and/or i.i.d.. See also the On the importance of the i.i.d. assumption in statistical learning thread.

As a sidenote, notice that we can even drop the assumption that each $Y_i$ comes from the same kind of distribution. There are (non-GLM) models that assume that different $Y_i$'s can have different distributions with different parameters, i.e. that your data comes from a mixture of different distributions. In such case we would also assume that the $Y_i$ values are independent, since dependent values, coming from different distributions with different parameters (i.e. typical real-world data) is something that in most cases would be too complicated to model (often impossible).

$\endgroup$
6
$\begingroup$

As has been stated, while we often consider the case of iid errors in linear regression, this does not have a direct equivalent in most generalized linear models (including logistic regression). In logistic regression, we typically employ the assumption of independence of outcomes that all have a very strict relation (i.e. linear effects on the log probabilities). But these result in random variables that are not identical, nor are they decomposable into a constant term plus an iid error as is the case with linear regression.

If you really want to show that the responses have some sort of iid relation, then follow me for the next paragraph. Just know that this idea is a little off the beaten path; you may not get full credit for this response on a final if your professor is lacking in patience.

You maybe familiar with the inverse-cdf method for generating random variables. If not, here's a refresher: if $X$ has cumulative distribution function $F_X$, then I can produce random draws from $X$ by first taking random draws $q \sim \text{uniform(0,1)}$ then calculating $X = F_X^{-1}(q)$. How does this relate to logistic regression? Well, we could think that the generating process for our responses has two parts; a fixed part relating the covariates to the probabilities of success, and a random part that determines the value of the random variable conditional on the fixed part. The fixed part is defined by the link function of logistic regression, i.e. $p = \text{expit}(\beta_o + \beta_1 x)$. For the random part, let's define $F_Y( y | p)$ to be the cdf for a Bernoulli distribution with probability $p$. Then we can think of the response variable $Y_i$ being generated by the following three steps:

1.) $p_i = \text{expit}(\beta_o + \beta_1 x_i)$

2.) $q_i \sim\text{uniform(0,1)}$

3.) $Y_i = F^{-1}(q_i | p_i)$

Then the standard assumption in logistic regression is that $q_i$ is iid.

$\endgroup$
  • 1
    $\begingroup$ You make several good points, but I'm not sure if talking about uniform i.i.d. $q_i$'s does not bring even more confusion. I would say that it's better to stick with the standard description that $Y_i \sim \mathcal{B}(p_i)$, that by itself assume that $Y_i$'s are Bernoulli-random with mean $p_i$. Defining it in terms of $q_i$'s makes it complicated because the "noise" is uniform, but then non-lineary transformed, so it gets ugly. $\endgroup$ – Tim Feb 3 '17 at 9:22
  • $\begingroup$ @Tim: yes, the second part of the answer is more of an interesting side note than a concise answer. But it may be a useful way to look at it; after all, that's basically how your computer simulates data from these models! $\endgroup$ – Cliff AB Feb 3 '17 at 14:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.