As part of self-study, I am reviewing arguments I found tricky from Larry Wasserman's course notes *"Intermediate Statistics Fall 2016, Lecture Notes 6: Likelihood"*. In particular, I have a number of queries concerning a heuristic argument to show that the interpretation of the likelihood function as "*that which contains all the information in the data*", is faulty.

**Context.**

Here is an extract of the argument:

> The likelihood function is a minimal sufficient statistic. That is, if we define the equivalence relation: $x^n \sim y^n$ when $L(\theta; x^n) \propto L(\theta; y^n)$, then the resulting partition is minimal sufficient.
>
> Does this mean that the likelihood function contains all the relevant information? Some people say yes it does. This is sometimes called the *the likelihood principle*. That is, the likelihood principle says that the likelihood function contains all the information in the data.
>
> This is FALSE. Here is a simple example to illustrate why. Let $\mathcal{C} = \{c_1, \dots, c_N\}$ be a finite set of constants. For simplicity, assume that $c_j \in \{0, 1\}$ (although this is not important). Let $\theta = (1 / N)  \sum^N_{j=1} c_j$. Suppose we want to estimate $\theta$. We proceed as follows. Let $S_1, \dots S_N \sim \text{Bernoulli}(\pi)$ where $\pi$ is known. If $S_i = 1$, you get to see $c_i$. Otherwise, you do not. (This is an example of survey sampling.) The likelihood function is
>
> $$\prod_i \pi^{S_i} (1 - \pi)^{1 - S_i}.$$
>
> The unknown parameter does not appear in the likelihood. In fact, there are no unknown parameters in the likelihood! The likelihood function contains no information at all.
> But we can estimate $\theta$. Let
>
> $$\hat{\theta} = \frac{1}{N \pi} \sum^N_{j=1} c_j S_j.$$
>
> Then $\mathbb{E}[\hat{\theta}] = \theta$, Hoeffding's inequality implies that
>
> $$\mathbb{P}(\vert \hat{\theta} - \theta \vert > \epsilon) \leq 2 \exp(-2n \epsilon^2 \pi^2).$$
>
> Hence, $\hat{\theta}$ is close to $\theta$ with high probability.
>
> **Summary: The minimal sufficient statistic has all the information you need to compute the likelihood. But that does not mean that all the information is in the likelihood.**

**Queries.**

1\. *The context in which unbiasedness is discussed here departs from the standard contexts in which I have seen it. Is the following computation correct*?

\begin{align}\mathbb{E}[\hat{\theta}] &= \frac{1}{N \pi} \mathbb{E}_{p(S_1, \dots, S_n; \pi)} \left[ \sum^N_{j=1} c_j S_j \right] \\
&= \frac{1}{N \pi} \sum^N_{j=1} c_j \mathbb{E}_{p(S_1, \dots, S_N ; \pi)} \left[ S_j \right] \\
&= \frac{\pi}{N \pi} \sum^N_{j=1} c_j \\
&= \theta.
&= \end{align}

2\. *How is the fact that we can estimate $\hat{\theta}$ in spite of $\theta$ not being present in the likelihood function, and the fact that $\hat{\theta}$ is close to $\theta$ with high probability, relevant to the argument*?

Whilst I understand that the likelihood function $\prod_i \pi^{S_i} (1 - \pi)^{1 - S_i}$ does not contain the parameter $\theta$, I fail to see the relevance and importance of the counterpoint in the argument, "...*but we can estimate $\theta$.*" I am therefore unable to appreciate how showing that $\hat{\theta}$ is close to $\theta$ with high probability via Hoeffding's inequality fits into the argument. 

3\. *Are there any further references which discuss this heuristic argument and the issues contained within more broadly?*

I ask because my instinct is that the claim that "*[the likelihood function contains all the information in the data] is FALSE*" may be more contentious than the author might be suggesting. And I am not sure if this is a pedagogical simplification, or if it's the author's personal position on a contentious issue in statisitics, or if this enjoys general consensus in the statistical community. Essentially, I would like to know how contentious this claim is, if at all.

----------

In response to well-founded doubts about the fidelity of transcription in the comments, the following is a screenshot of the extract:

[![enter image description here][1]][1]

There seems to be no functionality to upload attachments, as I only have a local copy of the entire notes for this topic, but I can endeavour to find a working link for the notes should this be requested.


  [1]: https://i.sstatic.net/1lKOI.png