On whether an error term exists in logistic regression (and its assumed distribution), I have read in various places that:
Can someone please clarify?
In linear regression observations are assumed to follow a Gaussian distribution with a mean parameter conditional on the predictor values. If you subtract the mean from the observations you get the error: a Gaussian distribution with mean zero, & independent of predictor values—that is errors at any set of predictor values follow the same distribution.
In logistic regression observations $y\in\{0,1\}$ are assumed to follow a Bernoulli distribution† with a mean parameter (a probability) conditional on the predictor values. So for any given predictor values determining a mean $\pi$ there are only two possible errors: $1-\pi$ occurring with probability $\pi$, & $0-\pi$ occurring with probability $1-\pi$. For other predictor values the errors will be $1-\pi'$ occurring with probability $\pi'$, & $0-\pi'$ occurring with probability $1-\pi'$. So there's no common error distribution independent of predictor values, which is why people say "no error term exists" (1).
"The error term has a binomial distribution" (2) is just sloppiness—"Gaussian models have Gaussian errors, ergo binomial models have binomial errors". (Or, as @whuber points out, it could be taken to mean "the difference between an observation and its expectation has a binomial distribution translated by the expectation".)
"The error term has a logistic distribution" (3) arises from the derivation of logistic regression from the model where you observe whether or not a latent variable with errors following a logistic distribution exceeds some threshold. So it's not the same error defined above. (It would seem an odd thing to say IMO outside that context, or without explicit reference to the latent variable.)
† If you have $k$ observations with the same predictor values, giving the same probability $\pi$ for each, then their sum $\sum y$ follows a binomial distribution with probability $\pi$ and no. trials $k$. Considering $\sum y -k\pi$ as the error leads to the same conclusions.
This has been covered before. A model that is constrained to have predicted values in $[0,1]$ cannot possibly have an additive error term that would make the predictions go outside $[0,1]$. Think of the simplest example of a binary logistic model -- a model containing only an intercept. This is equivalent to the Bernoulli one-sample problem, often called (in this simple case) the binomial problem because (1) all the information is contained in the sample size and number of events or (2) the Bernoulli distribution is a special case of the binomial distribution with $n=1$. The raw data in this situation are a series of binary values, and each has a Bernoulli distribution with unknown parameter $\theta$ representing the probability of the event. There is no error term in the Bernoulli distribution, there's just an unknown probability. The logistic model is a probability model.
To me the unification of logistic, linear, poisson regression etc... has always been in terms of specification of the mean and variance in the Generalized Linear Model framework. We start by specifying a probability distribution for our data, normal for continuous data, Bernoulli for dichotomous, Poisson for counts, etc...Then we specify a link function that describes how the mean is related to the linear predictor:
$g(\mu_i) = \alpha + x_i^T\beta$
For linear regression, $g(\mu_i) = \mu_i$.
For logistic regression, $g(\mu_i) = \log(\frac{\mu_i}{1-\mu_i})$.
For Poisson regression, $g(\mu_i) = \log(\mu_i)$.
The only thing one might be able to consider in terms of writing an error term would be to state:
$y_i = g^{-1}(\alpha+x_i^T\beta) + e_i$ where $E(e_i) = 0$ and $Var(e_i) = \sigma^2(\mu_i)$. For example, for logistic regression, $\sigma^2(\mu_i) = \mu_i(1-\mu_i) = g^{-1}(\alpha+x_i^T\beta)(1-g^{-1}(\alpha+x_i^T\beta))$. But, you cannot explicitly state that $e_i$ has a Bernoulli distribution as mentioned above.
Note, however, that basic Generalized Linear Models only assume a structure for the mean and variance of the distribution. It can be shown that the estimating equations and the Hessian matrix only depend on the mean and variance you assume in your model. So you don't necessarily need to be concerned with the distribution of $e_i$ for this model because the higher order moments don't play a role in the estimation of the model parameters.
