5
$\begingroup$

I have some questions on the difference between conditional MLE (CMLE) and unconditional MLE (UMLE) in practice. In what follows I will only talk about the unconditional and conditional mean and leave out the variance in order to shorten the question.

Example 1:

We have a linear model: $y_{i}=x_{i}^{\prime}\beta+\varepsilon_{i}$, $\varepsilon_{i}\sim N\left(0,\:\sigma^{2}\right)$

$y$ has conditional mean: $E\left[y\mid x\right]=x_{i}^{\prime}\beta$, and unconditional mean: $E\left[y\right]=\mu$

Am I right to assume that unconditional Maximum Likelihood estimation proceeds like this?

The density function for observation $y$ is: $f\left(y_{i}\right)=\left(\frac{1}{\sqrt{2\pi\sigma^{2}}}\right)\exp\left(-\frac{1}{2}\frac{\left(y_{i}-\mu\right)^{2}}{\sigma^{2}}\right)$

The log-likelihood function is:$\log L\left(\mu,\:\sigma^{2}\right)=-\frac{N}{2}\log\left(2\pi\sigma^{2}\right)-\frac{1}{2}\sum_{i=1}^{N}\frac{\left(y_{i}-\mu\right)^{2}}{\sigma^{2}}$

And the UMLE estimator is: $\hat{\mu}_{ML}=\frac{1}{N}\sum_{i=1}^{N}y_{i}$

Am I right to assume that conditional Maximum Likelihood estimation proceeds like this? $f\left(y_{i}\mid x_{i}\right)=\left(\frac{1}{\sqrt{2\pi\sigma^{2}}}\right)\exp\left(-\frac{1}{2}\frac{\left(y_{i}-x_{i}^{\prime}\beta\right)^{2}}{\sigma^{2}}\right)$

the log-likelihood function is: $\log L\left(\beta,\:\sigma^{2}\right)=-\frac{N}{2}\log\left(2\pi\sigma^{2}\right)-\frac{1}{2}\sum_{i=1}^{N}\frac{\left(y_{i}-x_{i}^{\prime}\beta\right)^{2}}{\sigma^{2}}$

The CMLE estimator is: $\hat{\beta}_{ML}=\frac{\sum_{i=1}^{N}x_{i}y_{i}}{\sum_{i=1}^{N}x_{i}^{2}}$

Example 2

We have a Poisson model:

$y$ has conditional mean $E\left[y_{i}\mid x_{i}\right]=\lambda_{i}=\exp\left(x_{i}^{\prime}\beta\right)$ and unconditional mean: $E\left[y\right]=\lambda$.

Am I right to assume that unconditional Maximum Likelihood estimation proceeds like this?

The density function for $y$ is: $f\left(y_{i}\right)=\frac{e^{-\lambda}\lambda^{y_{i}}}{y_{i}!}$

The log-likelihood function for is: $\log L\left(\lambda\right)=\sum_{i=1}^{N}log\left(\frac{e^{-\lambda}\lambda^{y_{i}}}{y_{i}!}\right)=\sum_{i=1}^{N}\left(-\lambda+y_{i}\log\left(\lambda\right)-\log\left(y_{i}\right)\right)$

And the UMLE estimator is: $\hat{\lambda}_{ML}=\frac{1}{N}\sum_{i=1}^{N}y_{i}$

Am I right to assume that conditional Maximum Likelihood estimation proceeds like this? $f\left(y_{i}\mid x_{i}\right)=\frac{e^{-\lambda}\lambda^{y_{i}}}{y_{i}!}=\frac{\exp\left(-\exp\left(x_{i}^{\prime}\beta\right)\right)\cdot\left[\exp\left(x_{i}^{\prime}\beta\right)\right]^{y_{i}}}{y_{i}!}$

The log-likelihood function is: $\log L\left(\lambda\right)=\sum_{i=1}^{N}\left(\frac{\exp\left(-\exp\left(x_{i}^{\prime}\beta\right)\right)\cdot\left[\exp\left(x_{i}^{\prime}\beta\right)\right]^{y_{i}}}{y_{i}!}\right)=\sum_{i=1}^{N}\left(-\exp\left(x_{i}^{\prime}\beta\right)+y_{i}x_{i}^{\prime}\beta-log\left(y_{i}!\right)\right) $

And the CMLE estimator is the solution to: $\sum_{i=1}^{N}\left(y_{i}-\exp\left(x_{i}^{\prime}\beta\right)\right)x_{i}^{\prime}=0$

Example 3

We have an AR(1) model: $y_{t}=\mu+\Theta y_{t}+\varepsilon_{t}$, $\varepsilon_{t}\sim N\left(0,\:\sigma^{2}\right)$

$y$ has conditional mean: $E\left[y\mid y_{t-1}\right]=\mu+\Theta y_{t}$, and unconditional mean: $E\left[y\right]=\frac{\mu}{1-\Theta}$

$y$ has conditional variance: $Var\left[y_{t}\mid y_{t-1}\right]=\sigma^{2}$ and unconditional variance: $Var\left[y_{t}\right]=\frac{\sigma^{2}}{1-\Theta^{2}}$

Am I right to assume that unconditional Maximum Likelihood estimation proceeds like this?

The unconditional density function for $y$ is: $f\left(y_{t}\right)=\left(\frac{1}{\sqrt{2\pi\frac{\sigma^{2}}{1-\Theta^{2}}}}\right)\exp\left(-\frac{1}{2}\frac{\left(y_{t}-\frac{\mu}{1-\Theta}\right)^{2}}{\frac{\sigma^{2}}{1-\Theta^{2}}}\right)$

and then I state the log-likelihood function and derive the estimators as above.

Am I right to assume that conditional Maximum Likelihood estimation proceeds like this?

The conditional density function for $y$ is: $f\left(y_{t}\mid y_{t-1}\right)=\left(\frac{1}{\sqrt{2\pi\sigma^{2}}}\right)\exp\left(-\frac{1}{2}\frac{\left(y_{i}-\mu-\Theta y_{t-1}\right)^{2}}{\sigma^{2}}\right)$

and then I proceed by stating the log-likelihood function and deriving the estimators as above.

My question is if what I have done above is correct? I know that the CMLE is right since I condition on x. The question is if the UMLE part is correct? I am quite sure that Example 3 is correct but I am not so sure Example 1 and Example 2 are correct.

Any help would be greatly appreciated.

$\endgroup$
5
  • $\begingroup$ An equation like $\sum_{i=1}^{N}\left(y_{i}-\exp\left(x_{i}^{\prime}\beta\right)\right)x_{i}'=0$ can be expanded and simplified. $\endgroup$
    – Glen_b
    Oct 1, 2014 at 12:01
  • $\begingroup$ @Glen_B Thanks for pointing that out. I have corrected it. $\endgroup$
    – Plissken
    Oct 1, 2014 at 12:55
  • $\begingroup$ Unfortunately I think you have taken an unjustified step in your simplification (you can't exchange exponentiation and summation). I intended something rather more basic, like $\overline{y_{i}x_{i}'}=\overline{\exp\left(x_{i}^{\prime}\beta\right)x_{i}'}$, (though you may prefer to leave it as a sum rather than divide through by $N$) $\endgroup$
    – Glen_b
    Oct 1, 2014 at 23:53
  • $\begingroup$ Yeah that's what I thought. I see what you mean but I deleted it and kept the original. Thank you. $\endgroup$
    – Plissken
    Oct 2, 2014 at 8:54
  • $\begingroup$ I think there is a problem in example 1. From yi=x′iβ+εi, εi∼N(0,σ2), we can't obtain y ~ N(mu, sigma^2) which is followed by x is random. And Var(y) = Var(x'β) + σ2. $\endgroup$
    – user141362
    Dec 6, 2016 at 14:11

1 Answer 1

0
$\begingroup$

The main reason for using conditional maximum likelihood is the resulting distribution. For Y|X ~N(x'B,Var(eps)) holds because the variation of Y only depends on the (normal) variation of the eps. As you know the correct assumption of the underlying density is a crucial point in MML estimation and hence, with unconditional MML you would run into problems here. Think about how the unconditional distribution of Y would look like if X is for instance N(3, Var(eps)). Y would have a mixture distribution that is not even normal anymore. The density function that you write for unconditional likelihood (example 1) is therefore wrong and moreover, the variance of Y is definitely not σ2. If you don't condition on X the variation of X needs to be accounted for in the variation of Y. Again, the variation of X needs to be considered if you are using unconditional MML.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.