I'm studying some notes that present examples of sufficiency:

Let $Y_1, \dots, Y_n$ be i.i.d. $N(\mu, \sigma^2)$. Note that $\sum_{i = 1}^n (y_i - \mu)^2 = \sum_{i = 1}^n (y_i - \bar{y})^2 + n(\bar{y} - \mu)^2$. Hence

$$\begin{align} L(\mu, \sigma; \mathbf{y}) &= \prod_{i = 1}^n \dfrac{1}{\sqrt{2\pi \sigma^2}}e^{-\frac{1}{2\sigma^2}(y_i - \mu)^2} \\ &= \dfrac{1}{(2\pi \sigma^2)^{n/2}}e^{-\frac{1}{2\sigma^2}\sum_{i = 1}^n (y_i - \bar{y})^2}e^{-\frac{1}{2\sigma^2}n(\bar{y} - \mu)^2} \end{align}$$

From Theorem 1, it follows that where $T(\mathbf{Y}) = (\bar{Y}, \sum_{i = 1}^n (Y_i - \bar{Y})^2)$ is a sufficient statistic for $(\mu, \sigma)$.

Theorem 1 is presented as follows:

A statistic $T(\mathbf{Y})$ is sufficient for $\theta$ if, and only if, for all $\theta \in \theta$

$$L(\theta; \mathbf{y}) = g(T(\mathbf{y}), \theta) \times h(\mathbf{y})$$

where the function $g(\cdot)$ depends on $\theta$ and the statistic $T(\mathbf{Y})$, while the function $h(\cdot)$ does not contain $\theta$.

Theorem 1 implies that if the likelihood $L(\theta; \mathbf{y})$ depends on the data only through $T(\mathbf{y})$, $T(\mathbf{Y})$ is a sufficient statistic for $\theta$ and $h(\mathbf{y}) \equiv 1$.

For reference to another example, here is a Poisson example that I recently posted:

Let $Y_1, \dots, Y_n$ be a i.i.d. $\text{Pois}(\lambda)$. Then

$$\begin{align} L(\lambda; \mathbf{y} &= \prod_{i = 1}^n e^{-\lambda} \dfrac{\lambda^{y_i}}{y_i!} \\ &= e^{-\lambda n} \dfrac{\lambda^{\sum_{i = 1}^n y_i}}{\prod_{i = 1}^n y_i!} \\ &= g(T(\mathbf{y}), \lambda) \times h(\mathbf{y}) \end{align}$$

where $T(\mathbf{y}) = \sum_{i = 1}^n y_i$, $g(T(\mathbf{y}), \lambda) = e^{-\lambda n} \lambda^{T(\mathbf{y})}$ and $h(\mathbf{y}) = \dfrac{1}{\prod_{i = 1}^n y_i!}$

There are three things that I don't understand here:

  1. How is it that $\sum_{i = 1}^n (y_i - \mu)^2 = \sum_{i = 1}^n (y_i - \bar{y})^2 + n(\bar{y} - \mu)^2$? EDIT: Answered here.

  2. If, for $L(\theta; \mathbf{y})$, we require the form $g(T(\mathbf{y}), \theta) \times h(\mathbf{y})$, then, for $L(\mu, \sigma; \mathbf{y})$, what form do we require? Trying to think of this myself, I thought of three potentially correct forms: $g(T(\mathbf{y}), (\mu, \sigma)) \times h(\mathbf{y})$, $g(T(\mathbf{y}), (\sigma, \mu)) \times h(\mathbf{y})$, or $g(T(\mathbf{y}), \mu, \sigma) \times h(\mathbf{y})$.

  3. Related to 2., comparing the first example to the Poisson example, I don't understand the conclusion of the first example. How does $T(\mathbf{Y}) = (\bar{Y}, \sum_{i = 1}^n (Y_i - \bar{Y})^2)$ satisfy the form $g(T(\mathbf{y}), \lambda) \times h(\mathbf{y})$?

I would greatly appreciate it if people would please take the time to clarify these points.


2 Answers 2

  1. You can check immediately that the vectors $$ \begin{bmatrix} y_1 - \bar{y}\\ \vdots \\ y_n - \bar{y}\\ \end{bmatrix}, \begin{bmatrix} \bar{y} -\mu\\ \vdots \\ \bar{y} -\mu\\ \end{bmatrix} $$ are orthogonal in $\mathbb{R}^n$.

  2. Follow what the theorem says: $\theta = (\mu, \sigma)$. So $$ g(T(\mathbf{y}), \theta) \times h(\mathbf{y}) $$ becomes $$ g(T(\mathbf{y}), (\mu, \sigma)) \times h(\mathbf{y}). $$

  3. Take $h(\mathbf{y}) = 1$.

The geometric meaning of your example is the following. The density of the multivariate normal distribution $$ \mathcal{N}( \mu \cdot \begin{bmatrix} 1\\ \vdots \\ 1\\ \end{bmatrix}, \sigma^2 I) $$ is constant on concentric spheres centered at $\begin{bmatrix} \mu\\ \vdots \\ \mu\\ \end{bmatrix}$. Now take an affine hyperplane $V$ in $\mathbb{R}^n$ that is orthogonal to line $L$ parametrized by $$ \mu \cdot \begin{bmatrix} 1\\ \vdots \\ 1\\ \end{bmatrix}, \;\; \mu \in \mathbb{R}. $$ Suppose $V$ intersects $L$ at the point $ \begin{bmatrix} a\\ \vdots \\ a\\ \end{bmatrix} $, then $y \in V$ if and only if its sample mean $\bar{y} = a$. Therefore the statistic $$ T(y) = (\bar{y} = a, \sum_{i = 1}^n (y_i - \bar{y})^2) $$ corresponds to an $n-1$-dimensional sphere lying in $V$ of fixed radius $\sqrt{\sum_{i = 1}^n (y_i - a)^2}$ centered at $ \begin{bmatrix} a\\ \vdots \\ a\\ \end{bmatrix}. \;\; $

It is clear from the geometry that, independent of $(\mu, \sigma^2)$, the density of any distribution in your family is constant on such $n-1$-dimensional spheres---therefore $h(\mathbf{y}) = 1$. This means that condition on the statistic $T(y)$, data is uniformly distributed independent of the parameter.

  • $\begingroup$ Thanks for the answer. How does your answer with regards to 3. answer the question? $\endgroup$ Apr 13, 2021 at 18:33

I'll try to make the answers a little bit more approachable. For your first question, you might take a brute-force approach: $$\begin{aligned} \sum (y_i - \mu)^2 &= \sum (y_i^2 - 2\mu y_i + \mu^2) \\ &=\sum (y_i^2 - 2\mu y_i + \mu^2 + \bar y^2 - \bar y^2 + 2\bar y y_i - 2\bar y y_i) \\ &= \sum\Big[(y_i - \bar y)^2 + \mu^2 - 2\mu y_i - \bar y^2 + 2\bar y y_i\Big] \\ &= \sum(y_i - \bar y)^2 + n\mu^2 - 2n\mu \bar y - n\bar y^2 + 2n \bar y^2 \\ &= \sum(y_i - \bar y)^2 + n(\mu^2 - 2\mu \bar y + \bar y^2) \\ &= \sum(y_i - \bar y)^2 + n(\mu - \bar y)^2 \end{aligned}$$ It's simple algebra.

To answer your second question: the order of the elements in the parameter vector are unimportant, i.e., we might treat $\theta$ as $[\mu, \sigma]$ or $[\sigma, \mu]$; it doesn't matter. So all three forms are right.

Lastly, for your third question, "take $h(y) = 1$" means that $$ g(T(y), \theta) = \frac{1}{(2\pi \sigma^2)^{n/2}}e^{-\frac{1}{2\sigma^2}\sum_{i = 1}^n (y_i - \bar{y})^2}e^{-\frac{1}{2\sigma^2}n(\bar{y} - \mu)^2}.$$

Notice that in the expression of $g(T(y), \theta)$, the only statistics that appear are $\bar y$ and $\sum_{i=1}^n (y_i - \bar y)$. The data interact, so to say, with the parameters only through these statistics. They are jointly sufficient.

  • $\begingroup$ Even if the two vectors $(\mu, \sigma)$ and $(\sigma, \mu)$ are equally valid, are you sure that $g(T(\mathbf{y}), \mu, \sigma) \times h(\mathbf{y})$ is the same as the other two $g(T(\mathbf{y}), (\mu, \sigma)) \times h(\mathbf{y})$ and $g(T(\mathbf{y}), (\sigma, \mu)) \times h(\mathbf{y})$? The first one has the two variables $\sigma$ and $\mu$ as elements of the function $g$, whereas the other two have the vectors $(\mu, \sigma)$ or $(\sigma, \mu)$ as an element of the function $g$. $\endgroup$ Apr 13, 2021 at 18:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.