I was trying to understand asymptotic normality of the posterior better, and came across a confusing point. So let's say we have a likelihood, $L(\theta | X) = \Pi_{i=1}^n p(X_i | \theta)$, so the log-likelihood is $J(\theta) = log L = \Sigma_{i=1}^n log(p(X_i | theta))$.

J is itself a sum of random variables, so the log-likelihood J will be asymptotically normal, by the central limit theorem.

But we can also show the likelihood is asymptotically normal through a Taylor expansion. Let $\hat{\theta}$ be the mle. So we have

$J(\theta) = J(\hat{\theta}) + \nabla J \cdot (\theta-\hat{\theta}) + \frac{1}{2}(\theta-\hat{\theta})H(\theta-\hat{\theta})$. Since $\hat{\theta}$ is the mle, we know $\nabla J = 0$, and $I(\theta)=-H$ so this reduces to

(1) $J(\theta) = log(L) = J(\hat{\theta}) - \frac{1}{2}(\theta-\hat{\theta})I(\theta)(\theta-\hat{\theta})$

Now exponentiating (1), we get

$e^J = L = ke^{-\frac{1}{2}(\theta-\hat{\theta})I(\theta)(\theta-\hat{\theta})}$, which is also asymptotically normal, with L ~ $N(\hat{\theta},I(\theta)^{-1})$.

Am I making a mistake here...?

  • 2
    $\begingroup$ If the log likelihood is asymptotically normal, then the likelihood must be asymptotically lognormal. Can it then at the same time be asymptotically normal? asymptotics can be strange ... $\endgroup$ – kjetil b halvorsen Feb 27 '19 at 21:12

I think you just have to be precise about what you mean by "asymptotically normal." For example, when people say that "a sum of random variables is asymptotically normal by the central limit theorem," they usually really mean a precise statement about convergence in distribution, e.g.,

Central Limit Theorem (Lindeberg–Lévy version). Suppose $(X_n)_{n=1}^\infty$ is a sequence of i.i.d. random variables with mean $\mu$ and variance $\sigma^2 < \infty$. Let $S_n = n^{-1}(X_1 + \cdots + X_n)$ (the $n$th sample mean). Then $$ \sqrt{n} (S_n - \mu) \Rightarrow N(0, \sigma^2) $$ as $n \to \infty$ (here $\Rightarrow$ denotes convergence in distribution).

This doesn't say that $S_n \Rightarrow N(\mu, \sigma^2/n)$ as $n \to \infty$, which is formally impossible because the expression on the right-hand side involves $n$, but it is often informally stated as $S_n \approx N(\mu, \sigma^2/n)$ for large $n$ (the symbol $\approx$ should be read "is approximately distributed as").

In your case, you have a sequence $(L_n)_{n=1}^\infty$ of log-likelihoods that, after appropriate standardization, become a sequence $(S_n)_{n=1}^\infty$ that satisfies $$ \sqrt{n}(S_n - \theta) \Rightarrow N(0, \sigma^2) $$ as $n \to \infty$ (for some $\theta$ and $\sigma^2$). Now you can recall the delta method:

Delta Method. Suppose $(S_n)_{n=1}^\infty$ is a sequence of random variables satisfying $$ \sqrt{n} (S_n - \theta) \Rightarrow N(0, \sigma^2) $$ as $n \to \infty$ for some constants $\theta$ and $\sigma^2$. Let $g : \mathbb{R} \to \mathbb{R}$ be a function such that $g^\prime(\theta)$ exists and is nonzero. Then $$ \sqrt{n}(g(S_n) - g(\theta)) \Rightarrow N(0, \sigma^2 \left(g^\prime(\theta)\right)^2) $$ as $n \to \infty$.

The hand-wavey interpretastion of this is that if $$ S_n \approx N(\theta, \sigma^2 / n) $$ for large $n$, then $$ g(S_n) \approx N(g(\theta), \sigma^2\left(g^\prime(\theta)\right)^2/n) $$ for large $n$ (provided that $g^\prime(\theta)$ exists and is nonzero).

In particular, it shouldn't be surprising that the sequences $(S_n)_{n=1}^\infty$ and $(\exp(S_n))_{n=1}^\infty$ are simultaneously "asymptotically normal."

| cite | improve this answer | |
  • 1
    $\begingroup$ This was super helpful, thanks for the reply. I missed the obvious delta method connection. Thanks. $\endgroup$ – user49404 Feb 27 '19 at 23:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.