# Should log-likelihood values increase when the sample size of a simulation increases?

If one simulates a process (such as an ARMA-GARCH process) with sample size $$n$$ and log-likelihood $$x$$, should this log-likelihood increase when the sample size increases to $$2n$$ for example?

## 1 Answer

It depends. More importantly though, it doesn't really matter.

Remember, in an iid setting, the Likelihood is the product of PDFs (or PMFs) as a function of $$\theta$$. If each $$f(x_i|\theta) < 1$$ then the Likelihood will get smaller for each additional point. Uniform distributions make this point clear.

Let $$X_1, \cdots X_n \sim U(0, \theta)$$, then clearly

$$L(\theta|{\bf X}) = \prod_{i=1}^n \frac{I(x_i < \theta)}{\theta} = \begin{cases} \frac{1}{\theta^{n}} & ,\max(x_1, \cdots x_n) \leq \theta \\ 0 & ,\max(x_1, \cdots x_n) >\theta \end{cases}$$ or if you prefer $$\log L(\theta|{\bf x}) = \begin{cases} -n\log(\theta) & ,\max(x_1, \cdots x_n) \leq\theta \\ -\infty & ,\max(x_1, \cdots x_n) > \theta \end{cases}$$ For values $$\theta < 1$$, the non-degenerate part of the likelihood gets arbitrarily big as $$n$$ increases. For value of $$\theta > 1$$, it gets arbitrarily small. But this doesn't really matter, since values of Likelihood aren't really interpretable. The important thing is that the Likelihood gets steeper around the likely values of $$\theta$$.

Here is a quick example.

Assume $$X_1, X_2, \cdots X_n \stackrel{iid}{\sim} N(\mu, 1)$$. Then \begin{align*}L(\mu|{\bf X}) &= \left(\frac{1}{\sqrt{2\pi}}\right)^n\exp\left\{\frac{-1}{2}\sum_{i=1}^n(x_i - \mu)^2 \right\} \\ &= \left(\frac{1}{\sqrt{2\pi}}\right)^n\exp\left\{\frac{-\sum_{i=1}^n x_i^2}{2}\right\}\exp\left\{n\mu (\bar{x} - \mu/2)\right\} \end{align*}

For illustration purposes, we set $$\bar{x} = 0$$ and $$\sum_{i=1}^n x_i^2 = n$$ (this is what we "expect" if $$\mu=0$$). Focusing on the vertical axis illustrates this point. The value of the likelihood itself doesn't really matter, it is the value of $$L(\theta_1|{\bf x})$$ relative to $$L(\theta_2|{\bf x})$$ that determines the likely values of $$\theta$$.

• For values $\theta<1$, the non-degenerate part of the likelihood gets arbitrarily big as n increases. For value of $\theta<1$, it gets arbitrarily small. One of the inequalities needs to be corrected. – Richard Hardy Dec 22 '18 at 20:06
• Thanks @knrumsey. While you do say likelihood values aren't really interprettable, surely they are comparable? For example, we can compare the aic's of non-nested models to see which has more plausible parameter estimates. Can we not also compare the likelihoods arising from estimating parameters with different sample sizes? – Vykta Wakandigara Dec 25 '18 at 7:20
• @VyktaWakandigara That is a good point! To some extent, Likelihood values can be compared. Of course we cannot always compare likelihoods directly, this is why AIC requires a penalty term to account for models with different number of parameters. Although I am not familiar with any such method, it is probably possible to do something similar for models using different sample sizes. Alternatively, a Bayesian approach might have some desirable features, since Posteriors probabilities can be interpretable where the Likelihood is not. – knrumsey Dec 29 '18 at 0:40