3
$\begingroup$

I feel like this is may be obvious, given similar questions and the responses given on this site. I'm still unclear as to the underlying latent variable specification with logistic regression though.

When estimating parameters in a logistic regression, I assumed that we define a model in terms of independent variables and coefficients. This model is transformed with the logit link function and entered as p in the joint product of Bernoulli pdfs, from which MLE is applied to estimate parameters.

I have been reading that there is an underlying latent variable that is defined as

$$y_i=x_i*beta+epsilon_i$$

Where epsilon has a logistic distribution and individual dependent variable Y is Y=1 when y>0 and Y=0 when y<0. I have a few questions around this.

  1. Why do we need this?
  2. Why does epsilon have a logistic distribution
  3. Why Y=1 when y>0 and vice versa
$\endgroup$

1 Answer 1

2
$\begingroup$
  1. Why do we need this?

A latent variable model is not always needed, but it can be useful as description of the underlying process that generates the observations.

An advantage of understanding the underlying process is that one may design experiments that use more direct measurements of the latent variables and improve the understanding and knowledge of the system. Also, sometimes theoretical considerations may help to estimate the parameters of the model.

  1. Why does epsilon have a logistic distribution
  2. Why Y=1 when y>0 and vice versa

This is just the specification for a particular type of latent variable model from which the logistic regression model follows.

With that latent variable model. The probability that $Y_i=1$, is equal to the probability that $y_i>0$ (note that low case and upper case are different variables here), which is equal to the probability that $\epsilon_i > -X_i\beta$, which is equal to the logistic function $1/(1+e^{-X_i\beta})$.

It is not the only way to get a logistic regression model. See: Deduce the logistic regression formula and How to prove that LDA has a similar form to Logistic Regression for the binary classification example?


The logistic distribution can also be used as an approximation to the normal distribution. A model with a normal distributed latent variable would lead to a (computationally more difficult) probit model. The Wikipedia page on the probit model has a good explanation for the underlying latent variable model which is analogous to the latent variable model for the logistic regression that you consider.

An example where this approximation is used is in the computation of ELO scores for chess (Does the Elo system work with different expected value functions?). So in that case the latent variable is not for some specific theoretical reason logistic distributed, and the reason to use it is because it makes computations easier.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.