I am still trying to learn (may be the terminology issue) what does "link function" mean. For example, in logistic regression, we assume response variable is coming form binomial distribution.

The $\text{logit}^{-1}$ link function convert a real number from $(-\infty, -\infty)$ (output from $\beta^{\top}x$) to a probability number $[0,1]$. But how does it "link" to a binomial distribution which is a discrete distribution?

I understand the "link" is between a real number to a probability number, but there is some missing part from probability number to binomial distribution.

Am I right?

  • 2
    $\begingroup$ You might find the detailed discussion of an unusual link function in my answer at stats.stackexchange.com/a/64039/919 to be of some interest. (The question very well may be a duplicate of yours.) There is nothing missing: the response in logistic regression is Bernoulli and it is completely determined by its parameter (your "probability number"). $\endgroup$ – whuber Feb 2 '17 at 21:55
  • $\begingroup$ thanks @whuber for helping me all the time. the link you provided is valuable but I will never find it because of the weird question title... $\endgroup$ – Haitao Du Feb 2 '17 at 21:57
  • $\begingroup$ Finding good search terms is always a problem--I'm not faulting you or anyone else for not finding it. (Whenever I need to find that post again, I search on "sunflower," of all things!) $\endgroup$ – whuber Feb 2 '17 at 21:58
  • 1
    $\begingroup$ See also Purpose of the link function in generalized linear model. $\endgroup$ – Scortchi - Reinstate Monica Feb 2 '17 at 22:00

So when you have binary response data, you have a "yes/no" or "1/0" outcome for each observation. However, what you are trying to estimate when doing a binary response regression is not a 1/0 outcome for each set of values of the independent variables you impose, but the probability that an individual with such characteristics will result in a "yes" outcome. Then the response is not discrete anymore, it's continuous (in the (0,1) interval). The response in the data (the true $y_i$) are, indeed, binary, but the estimated response (the $\Lambda(x_i'b)$ or $\Phi(x_i'b)$) are probabilities.

The underlying meaning of these link functions is that they are the distribution we impose to the error term in the latent variable model. Imagine each individual has an underlying (unobservable) willingness to say "yes" (or be a 1) in the outcome. Then we model this willingness as $y_i^*$ using a linear regression on the individual's characteristics $x_i$ (which is a vector in multiple regression):

$$y_i^*=x_i'\beta + \epsilon_i.$$

This is what is called a latent variable regression. If this individual's willingness was positive ($y_i^*>0$), the individual's observed outcome would be a "yes" ($y_i=1$), otherwise a "no". Note that the choice of threshold doesn't matter as the latent variable model has an intercept.

In linear regression we assume the error term to be normally distributed. In binary response and other models, we need to impose/assume a distribution on the error terms. The link function is the cumulative probability function that the error terms follow. For instance, if it is logistic (and we will use that the logistic distribution is symmetric in the fourth equality),

$$P(y_i=1)=P(y_i^*>0)=P(x_i'\beta + \epsilon_i>0)=P(\epsilon_i>-x_i'\beta)=P(\epsilon_i<x_i'\beta)=\Lambda(x_i'\beta).$$

If you assumed the errors to be normally distributed, then you would have a probit link, $\Phi(\cdot)$, instead of $\Lambda(\cdot)$.

| cite | improve this answer | |
  • $\begingroup$ +1 Welcome to our site, Anna! Thank you for contributing well-constructed answers in addition to the question you have asked. $\endgroup$ – whuber Feb 2 '17 at 22:24
  • $\begingroup$ Thanks! How did you see I was new? Is there something to track new people? Are you a moderator? I feel a little surprised. But, indeed, my intention was to give answers much more than ask questions, but I just happened to have a question. $\endgroup$ – Anna SdTC Feb 2 '17 at 22:47
  • $\begingroup$ There's a lot to this site, Anna. Get started by reviewing our help center. You can click through almost anything you see for more information. Users with a diamond icon after their names are moderators, but so are any users with sufficiently large reputations. For additional questions about how this site works, go to our meta pages. The (idiosyncratic) site search is useful, but targeted Google searches (include "site:stats.stackexchange.com") can be even more effective. And check out our chat room. $\endgroup$ – whuber Feb 2 '17 at 23:16
  • $\begingroup$ @AnnaSdTC no there is no tracking mechanism. There is a review queue that highlights posts by new users, but in most cases you can simply notice new nickname + avatar. Also in profile info there is an information on when the account was created (see yourelf stats.stackexchange.com/users/146969/anna-sdtc, there is a "member for" section). $\endgroup$ – Tim Feb 2 '17 at 23:19
  • 1
    $\begingroup$ I've been looking for the answer to "why sigmoid" for logistic regression for a while and this is by far the best answer. I'm surprised that not many ML books mention this and impose the logistic function out of the blue. The best I've seen talks about GLM but it imposes the "GLM form" out of the blue and use that as "justification", which doesn't really explain anything. The only way I can understand is via this thinking - assumption on the distribution of the error term, and I think it is the only real explanation without imposing anything $\endgroup$ – Logan Yang Apr 16 at 16:16

Generalized linear model is defined in terms of linear predictor

$$ \eta = X\beta $$

Next thing is probability distribution that describes conditional distribution of $Y$ and a link function $g$ that "provides the relationship between the linear predictor and the mean of the distribution function", since we are not predicting the values of $Y$ but rather conditional mean of $Y$ given predictors $X$, i.e.

$$ E(Y|X) = g^{-1}(\eta) $$

In case of Gaussian family GLM (linear regression) identity function is used as a link function, so $E(Y|X) = \eta$, while in case of logistic regression logit function is used. (Inverse of) logit function transforms values of $\eta$ in $(-\infty, \infty)$ to $(0, 1)$, since logistic regression predicts probabilities of success, i.e. mean of Bernoulli distribution. Other functions are used for transforming linear predictors to means of different distributions, for example log function for Poisson regression, or inverse link for gamma regression. So link function does not link values of $Y$ (e.g. binary, in case of logistic regression) and linear predictor, but mean of the distribution of $Y$ with $\eta$ (actually, to translate the probabilities to $0$'s and $1$'s you would additionally need a decition rule). So the take-away message is that we are not predicting the values of $Y$ but instead describing it in terms of probabilistic model and estimating parameters of conditional distribution of $Y$ given $X$.

For learning more about link functions and GLM's you can check Difference between 'link function' and 'canonical link function' for GLM, Purpose of the link function in generalized linear model and Difference between logit and probit models threads, the very good Wikipedia article on GLM's and the Generalized linear models book by McCullagh and Nelder.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.