6
$\begingroup$

If one simulates a process (such as an ARMA-GARCH process) with sample size $n$ and log-likelihood $x$, should this log-likelihood increase when the sample size increases to $2n$ for example?

$\endgroup$

1 Answer 1

9
$\begingroup$

It depends. More importantly though, it doesn't really matter.

Remember, in an iid setting, the Likelihood is the product of PDFs (or PMFs) as a function of $\theta$. If each $f(x_i|\theta) < 1$ then the Likelihood will get smaller for each additional point. Uniform distributions make this point clear.

Let $X_1, \cdots X_n \sim U(0, \theta)$, then clearly

$$L(\theta|{\bf X}) = \prod_{i=1}^n \frac{I(x_i < \theta)}{\theta} = \begin{cases} \frac{1}{\theta^{n}} & ,\max(x_1, \cdots x_n) \leq \theta \\ 0 & ,\max(x_1, \cdots x_n) >\theta \end{cases}$$ or if you prefer $$\log L(\theta|{\bf x}) = \begin{cases} -n\log(\theta) & ,\max(x_1, \cdots x_n) \leq\theta \\ -\infty & ,\max(x_1, \cdots x_n) > \theta \end{cases}$$ For values $\theta < 1$, the non-degenerate part of the likelihood gets arbitrarily big as $n$ increases. For value of $\theta > 1$, it gets arbitrarily small. But this doesn't really matter, since values of Likelihood aren't really interpretable. The important thing is that the Likelihood gets steeper around the likely values of $\theta$.

Here is a quick example.

Assume $X_1, X_2, \cdots X_n \stackrel{iid}{\sim} N(\mu, 1)$. Then \begin{align*}L(\mu|{\bf X}) &= \left(\frac{1}{\sqrt{2\pi}}\right)^n\exp\left\{\frac{-1}{2}\sum_{i=1}^n(x_i - \mu)^2 \right\} \\ &= \left(\frac{1}{\sqrt{2\pi}}\right)^n\exp\left\{\frac{-\sum_{i=1}^n x_i^2}{2}\right\}\exp\left\{n\mu (\bar{x} - \mu/2)\right\} \end{align*}

For illustration purposes, we set $\bar{x} = 0$ and $\sum_{i=1}^n x_i^2 = n$ (this is what we "expect" if $\mu=0$).Likelihoods for different values of n

Focusing on the vertical axis illustrates this point. The value of the likelihood itself doesn't really matter, it is the value of $L(\theta_1|{\bf x})$ relative to $L(\theta_2|{\bf x})$ that determines the likely values of $\theta$.

$\endgroup$
3
  • $\begingroup$ For values $\theta<1$, the non-degenerate part of the likelihood gets arbitrarily big as n increases. For value of $\theta<1$, it gets arbitrarily small. One of the inequalities needs to be corrected. $\endgroup$ Commented Dec 22, 2018 at 20:06
  • $\begingroup$ Thanks @knrumsey. While you do say likelihood values aren't really interprettable, surely they are comparable? For example, we can compare the aic's of non-nested models to see which has more plausible parameter estimates. Can we not also compare the likelihoods arising from estimating parameters with different sample sizes? $\endgroup$ Commented Dec 25, 2018 at 7:20
  • 1
    $\begingroup$ @VyktaWakandigara That is a good point! To some extent, Likelihood values can be compared. Of course we cannot always compare likelihoods directly, this is why AIC requires a penalty term to account for models with different number of parameters. Although I am not familiar with any such method, it is probably possible to do something similar for models using different sample sizes. Alternatively, a Bayesian approach might have some desirable features, since Posteriors probabilities can be interpretable where the Likelihood is not. $\endgroup$
    – knrumsey
    Commented Dec 29, 2018 at 0:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.