I notice that for many common distributions (e.g., Bernoulli, Exponential), the Fisher Information is the same as the reciprocal of the population variance.
When is this true? Is there a theorem that states than if conditions X, Y, and Z are true, then we have that $$ I(\theta ;X) = \frac{1}{\text{Var}(X)} $$
In both those cases (Bernoulli and exponential) the MLE estimator is the identity function. This gives a clue as to how to proceed. Here is a theorem giving sufficient conditions for this result.
Theorem: Consider a family of distributions $\{ F_\theta | \theta \in \Theta \}$. If the estimator $\hat{\theta}(x) = x$ (i.e., the identity estimator) is efficient, then we have:
$$\mathcal{I}(\theta) = \frac{1}{\mathbb{V}(X)}.$$
Proof: The variance of the identity estimator is $\mathbb{V}(\hat{\theta}) = \mathbb{V}(X)$. If the estimator is efficient then (by definition) it achieves the Cramér–Rao bound, so we then have:
$$\mathbb{V}(\hat{\theta}) = \frac{1}{\mathcal{I}(\theta)}.$$
Putting these two results together gives the required result. $\blacksquare$
