As part of self-study, I am reviewing arguments I found tricky from Larry Wasserman's course notes "Intermediate Statistics Fall 2016, Lecture Notes 6: Likelihood". In particular, I have a number of queries concerning a heuristic argument to show that the interpretation of the likelihood function as "that which contains all the information in the data", is faulty.
Context.
Here is an extract of the argument:
The likelihood function is a minimal sufficient statistic. That is, if we define the equivalence relation: $x^n \sim y^n$ when $L(\theta; x^n) \propto L(\theta; y^n)$, then the resulting partition is minimal sufficient.
Does this mean that the likelihood function contains all the relevant information? Some people say yes it does. This is sometimes called the the likelihood principle. That is, the likelihood principle says that the likelihood function contains all the information in the data.
This is FALSE. Here is a simple example to illustrate why. Let $\mathcal{C} = \{c_1, \dots, c_N\}$ be a finite set of constants. For simplicity, assume that $c_j \in \{0, 1\}$ (although this is not important). Let $\theta = (1 / N) \sum^N_{j=1} c_j$. Suppose we want to estimate $\theta$. We proceed as follows. Let $S_1, \dots S_N \sim \text{Bernoulli}(\pi)$ where $\pi$ is known. If $S_i = 1$, you get to see $c_i$. Otherwise, you do not. (This is an example of survey sampling.) The likelihood function is
$$\prod_i \pi^{S_i} (1 - \pi)^{1 - S_i}.$$
The unknown parameter does not appear in the likelihood. In fact, there are no unknown parameters in the likelihood! The likelihood function contains no information at all. But we can estimate $\theta$. Let
$$\hat{\theta} = \frac{1}{N \pi} \sum^N_{j=1} c_j S_j.$$
Then $\mathbb{E}[\hat{\theta}] = \theta$, Hoeffding's inequality implies that
$$\mathbb{P}(\vert \hat{\theta} - \theta \vert > \epsilon) \leq 2 \exp(-2n \epsilon^2 \pi^2).$$
Hence, $\hat{\theta}$ is close to $\theta$ with high probability.
Summary: The minimal sufficient statistic has all the information you need to compute the likelihood. But that does not mean that all the information is in the likelihood.
Queries.
1. The context in which unbiasedness is discussed here departs from the standard contexts in which I have seen it. Is the following computation correct?
\begin{align} \mathbb{E}[\hat{\theta}] &= \frac{1}{N \pi} \mathbb{E}_{p(S_1, \dots, S_n; \pi)} \left[ \sum^N_{j=1} c_j S_j \right] \\ &= \frac{1}{N \pi} \sum^N_{j=1} c_j \mathbb{E}_{p(S_1, \dots, S_N ; \pi)} \left[ S_j \right] \\ &= \frac{\pi}{N \pi} \sum^N_{j=1} c_j \\ &= \theta. \end{align}
2. How is the fact that we can estimate the parameter ${\theta}$ in spite of $\theta$ not being present in the likelihood function, and the fact that the estimator $\hat{\theta}$ is close to $\theta$ with high probability, relevant to the argument?
Whilst I understand that the likelihood function $\prod_i \pi^{S_i} (1 - \pi)^{1 - S_i}$ does not contain the parameter $\theta$, I fail to see the relevance and importance of the counterpoint in the argument, "...but we can estimate $\theta$." I am therefore unable to appreciate how showing that $\hat{\theta}$ is close to $\theta$ with high probability via Hoeffding's inequality fits into the argument.
3. Are there any further references which discuss this heuristic argument and the issues contained within more broadly?
I ask because my instinct is that the claim that "[the likelihood function contains all the information in the data] is FALSE" may be more contentious than the author might be suggesting. And I am not sure if this is a pedagogical simplification, or if it's the author's personal position on a contentious issue in statisitics, or if this enjoys general consensus in the statistical community. Essentially, I would like to know how contentious this claim is, if at all.
In response to well-founded doubts about the fidelity of transcription in the comments, the following is a screenshot of the extract:
There seems to be no functionality to upload attachments, as I only have a local copy of the entire notes for this topic, but I can endeavour to find a working link for the notes should this be requested.
