As part of self-study, I am reviewing arguments I found tricky from notes from an introductory theoretical statistics course by Larry Wasserman. In particular, I have a number of queries concerning a heuristic argument to show that the interpretation of the likelihood function as "that which contains all the information in the data", is faulty.

Context.

Here is an extract of the argument:

The likelihood function is a minimal sufficient statistic. That is, if we define the equivalence relation: $x^n \sim y^n$ when $L(\theta; x^n) \propto L(\theta; y^n)$, then the resulting partition is minimal sufficient.

Does this mean that the likelihood function contains all the relevant information? Some people say yes it does. This is sometimes called the the likelihood principle. That is, the likelihood principle says that the likelihood function contains all the information in the data.

This is FALSE. Here is a simple example to illustrate why. Let $\mathcal{C} = \{c_1, \dots, c_N\}$ be a finite set of constants. For simplicity, assume that $c_j \in \{0, 1\}$ (although this is not important). Let $\theta = (1 / N) \sum^N_{j=1} c_j$. Suppose we want to estimate $\theta$. We proceed as follows. Let $S_1, \dots S_n \sim \text{Bernoulli}(\pi)$ where $\pi$ is known. If $S_i = 1$, you get to see $c_i$. Otherwise, you do not. (This is an example of survey sampling.) The likelihood function is

$$\prod_i \pi^{S_i} (1 - \pi)^{1 - S_i}.$$

The unknown parameter does not appear in the likelihood. In fact, there are unknown parameters in the likelihood! The likelihood function contains no information at all. But we can estimate $\theta$. Let

$$\hat{\theta} = \frac{1}{N \pi} \sum^N_{j=1} c_j S_j.$$

Then $\mathbb{E}[\hat{\theta}] = \theta$, Hoeffding's inequality implies that

$$\mathbb{P}(\vert \hat{\theta} - \theta \vert > \epsilon) \leq 2 \exp(-2n \epsilon^2 \pi^2).$$

Hence, $\hat{\theta}$ is close to $\theta$ with high probability.

Summary: The minimal sufficient statistic has all the information you need to compute the likelihood. But that does not mean that all the information is in the likelihood.

Queries.

1. What is the "data", the distribution expectations are being computed with respect to, and the explicit steps to show $\hat{\theta}$ is unbiased?

Whilst I understand that $\hat{\theta}$ is unbiased, I am struggling to show why it is true. In particular, in the computation of $\mathbb{E}[\hat{\theta}]$. I am uncertain as to what the "data" is here, and as to what the distribution expectations are being computed with respect to is. Therefore I am unsure as to how to proceed in this calculation:

$$\mathbb{E}[\hat{\theta}] = \frac{1}{N \pi} \mathbb{E} \left[ \sum^N_{j=1} c_j S_j \right]$$

I am guessing that independence of $c_j$ and $S_j$ can be assumed, and I know the above should be equal to $\theta$.

2. How is the fact that we can estimate $\hat{\theta}$ in spite of $\theta$ not being present in the likelihood function, and the fact that $\hat{\theta}$ is close to $\theta$ with high probability, relevant to the argument?

Whilst I understand that the likelihood function $\prod_i \pi^{S_i} (1 - \pi)^{1 - S_i}$ does not contain the parameter $\theta$, I fail to see the relevance and importance of the counterpoint in the argument, "...but we can estimate $\theta$." I am therefore unable to appreciate how showing that $\hat{\theta}$ is close to $\theta$ with high probability via Hoeffding's inequality fits into the argument.

3. Are there any further references which discuss this heuristic argument and its the issues contained within more broadly?

I ask because my instinct is that the claim that "[the likelihood function contains all the information in the data] is FALSE" may be more contentious than the author might be suggesting. And I am not sure if this is a pedagogical simplification, or if it's the author's personal position on a contentious issue in statisitics, or if this enjoys general consensus in the statistical community. Essentially, I would like to know how contentious this claim is, if at all.