likelihood as random function wrt data

Suppose we have some dataset $$x= \{ x_1, \dots, x_n\}$$ where every datapoint is i.i.d., $$x_i \sim P(\cdot|\theta^*)$$ for some known distribution $$P$$ and true parameter $$\theta^*$$.

Then, for this dataset $$x$$, the likelihood is $$\mathcal{L}(\theta)= P(x|\theta) = \prod_i P(x_i|\theta)$$ is a function of parameters $$\theta$$. We can call this $$\mathcal{L}_x(\theta)$$ for that dataset $$x$$.

This likelihood function takes different forms depending on the data $$x$$. How do you characterize the distribution of $$\mathcal{L}_x(\theta)$$ in function space?

I know the mean of $$\mathcal{L}_x(\theta)$$ is approximately equal to $$\mathcal{L}_{\mu_x}(\theta)$$ via a first order Taylor expansion. What about the variance, and other properties of the distribution of the likelihood?

I'm not very deep in statistics, and don't really understand the role of moments and MGFs. How do these play a role (is it at all analogous to higher order terms in the Taylor expansion of a distribution?)

TLDR: What's the connection between the distribution of data, $$P(\cdot | \theta^*)$$, and the distribution of the likelihood function, $$P(x|\theta)$$ for some data $$x$$ coming from $$P(\cdot | \theta^*)$$?

The likelihood function $$L(\theta|X)$$ is indeed a random function (since $$X$$ is random) in the set of likelihood functions $$\{L(\cdot|x);\ x\in\mathsf X\}$$ Its distribution depends (obviously) on the statistical model. For instance
1. if $$L(\theta|X)$$ is the likelihood function attached to a Normal $$\mathcal N(\theta,1)$$ $$n$$-sample, and if $$\theta^*$$ is the true value of $$\theta$$, then $$-2\log(L(\theta^*|X_{1:n})) - \log(2\pi) = \sum_{i=1}^n (X_i-\theta^*)^2 \sim \chi^2_n$$ and $$-2\log[L(\theta|X_{1:n})] - \log(2\pi) = \sum_{i=1}^n (X_i-\theta)^2 \sim \chi^2_n(n(\theta-\theta^*)^2)$$
2. if $$L(\theta|X)$$ is the likelihood function attached to a Poisson $$\mathcal P(\theta)$$ distribution, and if $$\theta^*$$ is the true value of $$\theta$$, $$\log[L(\theta|X_{1:n})] = \log(\theta) \sum_{i=1}^n X_i -\theta -\sum_{i=1}^n \log (X_i!)$$ meaning that $$\left\{\log[L(\theta|X_{1:n})]+\sum_{i=1}^n \log (X_i!) +\theta\right\}\big/\log(\theta) \sim \mathcal P(n\theta^*)$$
• For a given value of $\theta$, $\log L(\theta|X_{1:n})$ is a sum of iid rvs, hence the CLT should apply under some appropriate conditions. – Xi'an Feb 10 at 16:37
• Ah thanks! So does that mean we can view the log likelihood as a gaussian process? (If the distribution of log $L_x(\theta)$ for a fixed $\theta$ is Gaussian, that would mean every joint and marginal of log $L_x(\theta)$ for different $\theta$ would be Gaussian?) – 900edges Feb 10 at 20:53