If one simulates a process (such as an ARMA-GARCH process) with sample size $n$ and log-likelihood $x$, should this log-likelihood increase when the sample size increases to $2n$ for example?


It depends. More importantly though, it doesn't really matter.

Remember, in an iid setting, the Likelihood is the product of PDFs (or PMFs) as a function of $\theta$. If each $f(x_i|\theta) < 1$ then the Likelihood will get smaller for each additional point. Uniform distributions make this point clear.

Let $X_1, \cdots X_n \sim U(0, \theta)$, then clearly

$$L(\theta|{\bf X}) = \prod_{i=1}^n \frac{I(x_i < \theta)}{\theta} = \begin{cases} \frac{1}{\theta^{n}} & ,\max(x_1, \cdots x_n) \leq \theta \\ 0 & ,\max(x_1, \cdots x_n) >\theta \end{cases}$$ or if you prefer $$\log L(\theta|{\bf x}) = \begin{cases} -n\log(\theta) & ,\max(x_1, \cdots x_n) \leq\theta \\ -\infty & ,\max(x_1, \cdots x_n) > \theta \end{cases}$$ For values $\theta < 1$, the non-degenerate part of the likelihood gets arbitrarily big as $n$ increases. For value of $\theta > 1$, it gets arbitrarily small. But this doesn't really matter, since values of Likelihood aren't really interpretable. The important thing is that the Likelihood gets steeper around the likely values of $\theta$.

Here is a quick example.

Assume $X_1, X_2, \cdots X_n \stackrel{iid}{\sim} N(\mu, 1)$. Then \begin{align*}L(\mu|{\bf X}) &= \left(\frac{1}{\sqrt{2\pi}}\right)^n\exp\left\{\frac{-1}{2}\sum_{i=1}^n(x_i - \mu)^2 \right\} \\ &= \left(\frac{1}{\sqrt{2\pi}}\right)^n\exp\left\{\frac{-\sum_{i=1}^n x_i^2}{2}\right\}\exp\left\{n\mu (\bar{x} - \mu/2)\right\} \end{align*}

For illustration purposes, we set $\bar{x} = 0$ and $\sum_{i=1}^n x_i^2 = n$ (this is what we "expect" if $\mu=0$).Likelihoods for different values of n

Focusing on the vertical axis illustrates this point. The value of the likelihood itself doesn't really matter, it is the value of $L(\theta_1|{\bf x})$ relative to $L(\theta_2|{\bf x})$ that determines the likely values of $\theta$.

  • $\begingroup$ For values $\theta<1$, the non-degenerate part of the likelihood gets arbitrarily big as n increases. For value of $\theta<1$, it gets arbitrarily small. One of the inequalities needs to be corrected. $\endgroup$ – Richard Hardy Dec 22 '18 at 20:06
  • $\begingroup$ Thanks @knrumsey. While you do say likelihood values aren't really interprettable, surely they are comparable? For example, we can compare the aic's of non-nested models to see which has more plausible parameter estimates. Can we not also compare the likelihoods arising from estimating parameters with different sample sizes? $\endgroup$ – Vykta Wakandigara Dec 25 '18 at 7:20
  • $\begingroup$ @VyktaWakandigara That is a good point! To some extent, Likelihood values can be compared. Of course we cannot always compare likelihoods directly, this is why AIC requires a penalty term to account for models with different number of parameters. Although I am not familiar with any such method, it is probably possible to do something similar for models using different sample sizes. Alternatively, a Bayesian approach might have some desirable features, since Posteriors probabilities can be interpretable where the Likelihood is not. $\endgroup$ – knrumsey Dec 29 '18 at 0:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.