2
$\begingroup$

From this page:

The bottom section : Summary of advantages of GLMs over traditional (OLS) regression contains the advantage

  • If the link produces additive effects, then we do not need constant variance.

Could someone explain what this means in the context of logistic regression please.

For example, suppose that the model was

$$ \log ( \text{odds} ) = \beta_0 + \beta_1 x_1 $$

where $x_1$ is a categorical variable with two levels.

The "link" function is the logit, which is the log of the odds.

What I don't understand is why being able to write the model in the above form means that we "do not need constant variance".

$\endgroup$
  • $\begingroup$ When I'm teaching logistic regression I don't spend time on variances but instead just on getting the model right in terms of it having a reasonable chance of fitting the data at hand. The logit link is loved because it places no restriction of the $\beta$s. $\endgroup$ – Frank Harrell Apr 8 at 11:53
2
$\begingroup$

This boils down to using links besides the canonical link for a non-Gaussian family. As the course notes say:

The choice of link is separate from the choice of random component thus we have more flexibility in modeling

You are not wrong to be confused by this, because nowhere in the introduction is one of these non-canonical links used. In the case of logistic regression, this means that we would not model the log odds, but rather have something called a linear probability model. This all would be a lot less confusing if we were to call it bernoulli regression, to emphasize the distribution of the response, conditional on $x$, rather than the connection between the linear predictor: $\eta = \beta_0 + \beta_1 x_i$ and the expected value of the response, $E(Y | \eta)$.

If we used a linear probability model we would have these three conditions hold: $$ \begin{align*} \eta & = \beta_0 + \beta_1 x, \\ E(Y|\eta) &= \eta, \\ \text{Var}(Y|\eta) & = E(Y|\eta)(1 - E(Y|\eta)). \end{align*} $$

The first equation holds for all generalized linear models. If we had the logit link, the second equation would read $\text{logit} \; E(Y|\eta) = \eta$ but the identity link means we don't put anything special here. (Some other non-canonical links include the log link and the complimentary log-log.) The third equation holds because we have the Bernoulli (aka binomial with 1 trial) family, and in this distribution the variance is always an inverted quadratic function of the mean. The notes are pointing out that line 2 can occur with just about any family, which would also be true with ordinary least squares, but we can still have something special happening with the conditional variance of $Y$.

$\endgroup$
2
$\begingroup$

What I don't understand is why being able to write the model in the above form means that we "do not need constant variance".

Suppose that we consider binary responses $Y_i\in\{0,1\}$. Then the mean given the covariates is a probability $\text E(Y_i \mid \vec x_i)=\pi_i(\vec x_i)$. Moreover, the variance of a binary variable is $\text{Var}(Y_i\mid\vec x_i)=\pi_i(\vec x_i)(1 - \pi_i(\vec x_i))$. Thus, since a generalized linear model is a model for the mean of the form (last equality is your example with a slope and an intercept)

$$ g(\pi_i(\vec x_i)) = g(\text E(Y_i\mid\vec x_i)) = \beta^\top\vec x_i = \beta_0 + \beta x_{i1} $$

then in this model the variance, $\pi_i(\vec x_i)(1 - \pi_i(\vec x_i))$, depends on the value $\vec x_i$ through

$$\pi_i(\vec x_i) = g^{-1}(\beta_0 + \beta x_{i1})$$

This is unlike a linear model where we assume that the variance is fixed.

The

If the link produces additive effects, then we ...

part may be slightly confusing. Even if we had non-linear regression model of the form

$$E(Y_i \mid \vec x_i) = \vec \beta^\top \vec h(\vec x_i) + \epsilon_i, \qquad \epsilon_i \sim N(0, \sigma²)$$

for some function $\vec h$ (e.g., a spline) then the variance would still be constant so the statement is kind of confusing. The main point is that generalized linear models have non-constant variance which depends on the mean (and potentially dispersion parameters).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.