Understand Link Function in Generalized Linear Model

Question

I am still trying to learn (may be the terminology issue) what does "link function" mean. For example, in logistic regression, we assume response variable is coming form binomial distribution.

The $\text{logit}^{-1}$ link function convert a real number from $(-\infty, -\infty)$ (output from $\beta^{\top}x$) to a probability number $[0,1]$. But how does it "link" to a binomial distribution which is a discrete distribution?

I understand the "link" is between a real number to a probability number, but there is some missing part from probability number to binomial distribution.

Am I right?

Answer 1

So when you have binary response data, you have a "yes/no" or "1/0" outcome for each observation. However, what you are trying to estimate when doing a binary response regression is not a 1/0 outcome for each set of values of the independent variables you impose, but the probability that an individual with such characteristics will result in a "yes" outcome. Then the response is not discrete anymore, it's continuous (in the (0,1) interval). The response in the data (the true $y_i$) are, indeed, binary, but the estimated response (the $\Lambda(x_i'b)$ or $\Phi(x_i'b)$) are probabilities.

The underlying meaning of these link functions is that they are the distribution we impose to the error term in the latent variable model. Imagine each individual has an underlying (unobservable) willingness to say "yes" (or be a 1) in the outcome. Then we model this willingness as $y_i^*$ using a linear regression on the individual's characteristics $x_i$ (which is a vector in multiple regression):

$$y_i^*=x_i'\beta + \epsilon_i.$$

This is what is called a latent variable regression. If this individual's willingness was positive ($y_i^*>0$), the individual's observed outcome would be a "yes" ($y_i=1$), otherwise a "no". Note that the choice of threshold doesn't matter as the latent variable model has an intercept.

In linear regression we assume the error term to be normally distributed. In binary response and other models, we need to impose/assume a distribution on the error terms. The link function is the cumulative probability function that the error terms follow. For instance, if it is logistic (and we will use that the logistic distribution is symmetric in the fourth equality),

$$P(y_i=1)=P(y_i^*>0)=P(x_i'\beta + \epsilon_i>0)=P(\epsilon_i>-x_i'\beta)=P(\epsilon_i<x_i'\beta)=\Lambda(x_i'\beta).$$

If you assumed the errors to be normally distributed, then you would have a probit link, $\Phi(\cdot)$, instead of $\Lambda(\cdot)$.

Answer 2

Generalized linear model is defined in terms of linear predictor

$$ \eta = X\beta $$

Next thing is probability distribution that describes conditional distribution of $Y$ and a link function $g$ that "provides the relationship between the linear predictor and the mean of the distribution function", since we are not predicting the values of $Y$ but rather conditional mean of $Y$ given predictors $X$, i.e.

$$ E(Y|X) = g^{-1}(\eta) $$

In case of Gaussian family GLM (linear regression) identity function is used as a link function, so $E(Y|X) = \eta$, while in case of logistic regression logit function is used. (Inverse of) logit function transforms values of $\eta$ in $(-\infty, \infty)$ to $(0, 1)$, since logistic regression predicts probabilities of success, i.e. mean of Bernoulli distribution. Other functions are used for transforming linear predictors to means of different distributions, for example log function for Poisson regression, or inverse link for gamma regression. So link function does not link values of $Y$ (e.g. binary, in case of logistic regression) and linear predictor, but mean of the distribution of $Y$ with $\eta$ (actually, to translate the probabilities to $0$'s and $1$'s you would additionally need a decition rule). So the take-away message is that we are not predicting the values of $Y$ but instead describing it in terms of probabilistic model and estimating parameters of conditional distribution of $Y$ given $X$.

For learning more about link functions and GLM's you can check Difference between 'link function' and 'canonical link function' for GLM, Purpose of the link function in generalized linear model and Difference between logit and probit models threads, the very good Wikipedia article on GLM's and the Generalized linear models book by McCullagh and Nelder.