1
$\begingroup$

Let $X_1,...,X_N$ be IID random variables sampled from a parametrised distribution $p_\theta$, and suppose my goal is to retrieve $\theta$ from these samples.

We know that the MLE provides an efficient unbiased estimator in the asymptotic regime, meaning it will be asymptotically unbiased, and have variance (asymptotically) saturating the Cramer-Rao bound.

On the other hand, we also know that even in the single-shot regime, that is, after a single observation, we can find an efficient locally unbiased estimator, with the major caveat of this estimator being in general only local (meaning it hinges on some prior knowledge about $\theta$). By contrast, the MLE is unbiased and efficient (albeit only asymptotically) without requiring prior knowledge of the parameter to estimate.

To concretise the discussion with an explicit toy example, say we want to estimate $p^2$ from a Bernoulli process, $X_i\sim\operatorname{Bern}(p)$. Standard calculations will then show that the MLE for this is $$\hat p_N^2 = \left(\frac{\sum_{i=1}^N X_i}{N}\right)^2, \\ \mathbb{E}[\hat p_N] = p^2 + \frac{p(1-p)}{N}, \\ \operatorname{Var}[\hat p_N] = \frac{4p^3(1-p)}{N} + O(1/N^2).$$ On the other hand, the single-shot estimator saturating the Cramer-Rao bound, locally around a parameter value $p^2$, is $$\hat f_{p^2}(X) = -p^2 + 2p X.$$ The calculations for the MLE are shown e.g. here, while those for the locally efficient estimator here. These two estimators thus have asymptotically the same variance (taking the average of the values of the local estimators for large $N$), but the MLE has the big advantage of being nonlocal, i.e. not a function of the parameter to estimate.

My question is: is there any intuition to understand why this should happen? Being that we're talking about IID variables, there shouldn't be more information in a sequence of observations than in each one individually. If $X_i,X_j$ were correlated I'd understand the need to use an estimator that operates on the full statistic at the same time, but why does this turn out to be useful when individual samples are IID?

$\endgroup$

1 Answer 1

0
$\begingroup$

One way to think about it is that if you use the other $N-1$ observations to estimate your prior $p$ in the local one-shot estimator, you will get an estimator that works globally and looks a lot like the MLE.

$\endgroup$
1
  • $\begingroup$ so you're saying, using multple shots we manage to get an estimator that is global while remaining (close to) efficient? That makes sense, but how would it actually work? $\endgroup$
    – glS
    Commented Apr 18 at 22:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.