1
$\begingroup$

What is the Likelihood function of a linear probability model?

I know the likelihood function is the joint probability density, but how to construct the likelihood function when we only have the probability $P(Y_i=1|X_i)$ and $P(Y_i=0|X_i)$?

$\endgroup$
3
  • 1
    $\begingroup$ If the likelihood is the joint density of $X_i$ and $Y_i$, then you need to know $P(X_i)$ in order to get the joint density. Whether $P(X_i)$ is meaningful or not is something you need to consider. $\endgroup$ Dec 14, 2011 at 12:43
  • 1
    $\begingroup$ What is a linear probability model? $\endgroup$
    – Xi'an
    Dec 14, 2011 at 14:21
  • $\begingroup$ The linear probability model is just linear regression with a binary outcome variable. $\endgroup$
    – Charlie
    Dec 14, 2011 at 15:48

2 Answers 2

6
$\begingroup$

In your example you simply don't have a likelihood function, because you defined just a probability model rather than a statistical one. If you know the probabilities $$ P(Y_i = 1 | X_i) = p_i \quad \mbox{and} \quad P(Y_i =0| X_i) = 1-p_i, $$ then you have a "unique" conditional probability model. First you have to understand the differences between probability and statistical models. Please, see this post.

In order to have a statistical model, the above probabilities must be unknown. Typically, some relation is imposed: $$ P_\theta(Y_i = 1 | X_i=x_i) = p_i(\theta),$$ where $\theta$ is the unknown parameter vector and $p_i(\theta) \in [0,1]$ for $i=1, \ldots, n$. Now, we have a parametric statistical model, since for each $\theta$ we have a probability model.

The likelihood function is $$ L(\theta) = \prod_{i=1}^n P_\theta(Y_i = y_i | X_i=x_i) = \prod_{i=1}^n p_i(\theta)^{y_i}(1-p_i(\theta))^{1-y_i}. $$

Usually, the shape of $p_i(\theta)$ is commonly specified as:

  1. $p_i(\theta) = \frac{\exp(\eta_i(\theta))}{1 + \exp(\eta_i(\theta))},$
  2. $p_i(\theta) = F(\eta_i(\theta)),$

where $\eta_i(\theta) = \alpha + \beta x_i$ and $F$ is a cumulative distribution function. You can find a good shape for $p_i$ by looking at the data behavior (plots, data dispersion and so forth).

$\endgroup$
1
  • 2
    $\begingroup$ +1. Note that by definition the Linear Probability Model (LPM) is case (2) with $F$ the identity function on the interval $[0,1]$ (and usually taken to be undefined on its complement). Alternatively, rather than making it undefined, one could use the CDF for a Uniform$(0,1)$ variable, in which case (2) includes the LPM. $\endgroup$
    – whuber
    Oct 7, 2018 at 14:46
2
$\begingroup$

The linear probability model (LPM) is $$\mathbb{P}(Y_i=1|X_i)=\beta_0+\beta_1X_i$$ with $Y_i$ a dummy random variable, and $X_i$ a random variable. If $X_i$ is instead a vector of random variables, the LPM is $$\mathbb{P}(Y_i=1|X_i)=\beta_0+X'_i\beta.$$ But let me continue with the simple case where $X_i$ is a random variable. It follows that $$\mathbb{P}(Y_i=0|X_i)=1-\mathbb{P}(Y_i=1|X_i)=1-\beta_0-\beta_1X_i.$$ Thus, for $y_i\in\{0,1\}$,$$\begin{align*}\mathbb{P}(Y_i=y_i|X_i)&=\mathbb{P}(Y_i=1|X_i)^{y_i}\mathbb{P}(Y_i=0|X_i)^{1-y_i}\\&=(\beta_0+\beta_1X_i)^{y_i}(1-\beta_0-\beta_1X_i)^{1-y_i}.\end{align*}$$ Now, if $\{(Y_i, X_i)\}_{i=1}^n$ is a random sample of $n\geq 1$ observations, the likelihood function is $$\begin{align*}\mathbb{P}(Y_1=y_1,Y_2=y_2,\ldots,Y_n=y_n|X_1,X_2,\ldots,X_n)&=\prod_{i=1}^n\mathbb{P}(Y_i=y_i|X_i)\\&=\prod_{i=1}^n(\beta_0+\beta_1X_i)^{y_i}(1-\beta_0-\beta_1X_i)^{1-y_i}.\end{align*}$$ Note that since probabilities are in $[0,1]$ we assume that $\beta_0+\beta_1 X_i$ is in $[0,1]$ for $i=1,\ldots,n$. In practice difficulties may occur when fitting the LPM because during the fitting process, fitted values may fall outside $[0,1]$ for some observed $X_i=x_i$.

$\endgroup$
1
  • $\begingroup$ This likelihood function did not include the needed constraints on the parameters to require probabilities to be in [0,1]. $\endgroup$ Oct 7, 2018 at 11:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.