How to prove the sufficiency of the least squares estimator?

Let $Y|X=Y_{1},...,Y_{n}$ a sequence of i.i.d random variables conditionated on the values of (fixed) $\mathbf{X}$, where \begin{equation} \mathbf{X}=\left[\begin{array}{cccc} 1 & x_{11} & ... & x_{k1} \\ 1 & x_{12} & ... & x_{k2} \\ \vdots & \vdots & \ddots & \vdots \\ 1 & x_{1n} & ... & x_{kn} \end{array}\right] \end{equation} The least squares estimator is defined by \begin{equation} \boldsymbol{\hat{\beta}}=(\mathbf{X'X})^{-1}\mathbf{X'}\mathbf{Y}\sim \mathcal{N}(\boldsymbol\beta,\boldsymbol\Omega=\sigma^{2}(X'X)^{-1}) \end{equation}

Then the densify function of the estimator is \begin{equation} f_{\hat{\boldsymbol\beta}}(\hat{\boldsymbol\beta}|\boldsymbol\beta,\boldsymbol\Omega)=\frac{1}{(2\pi)^{p/2}|\boldsymbol\Omega|^{1/2}}\exp\left\{-\frac{1}{2}(\boldsymbol{\hat{\beta}}-\boldsymbol\beta)'\boldsymbol\Omega^{-1}(\boldsymbol{\hat{\beta}}-\boldsymbol\beta)\right\} \end{equation}

and the likelihood of the sample is

\begin{align} f_{Y|X,\boldsymbol{\beta}\sigma^{2}I_{k}}(Y|X,\boldsymbol{\beta},\sigma^{2}I_{k})=\left(\frac{1}{2\pi\sigma^{2}}\right)^{n/2}\exp\left\{-\frac{1}{2\sigma^{2}}(Y-X\boldsymbol{\beta})'(Y-X\boldsymbol{\beta})\right\} \end{align}

I know that the factorization theorem states that

Let $X_{1}, X_{2}, ..., X_{n}$ denote random variables with joint probability density function $f(x_{1}, x_{2}, ..., x_{n}; θ)$, which depends on the parameter $θ$.

Then, the statistic $Y=u(X1,X2,...,Xn)$

is sufficient for θ if and only if the p.d.f can be factored into two components, that is:

\begin{equation} f(x_{1},x_{2},...,x_{n};θ)=\phi[u(x_{1},x_{2},...,x_{n});θ]h(x1,x2,...,xn) \end{equation}

How to proceed in this case?

  • 1
    $\begingroup$ This deserves a self-study tag, I believe, and you have to detail why you cannot establish the sufficiency on your own. $\endgroup$
    – Xi'an
    Aug 26, 2016 at 0:07
  • 1
    $\begingroup$ Okay. I will edit the question. $\endgroup$ Aug 26, 2016 at 0:09
  • $\begingroup$ What do you know about sufficiency? What have you tried? $\endgroup$
    – Glen_b
    Aug 26, 2016 at 0:42
  • $\begingroup$ @Glen_b I have edited the question $\endgroup$ Aug 26, 2016 at 0:48

1 Answer 1


Starting with the joint density (/ likelihood)

  1. try expanding the exponent, collecting terms (in the parameter, so you get terms of the form $\beta'A\beta$ and $B\beta$ for example )

  2. "complete the square", rewriting the terms in the form $(\beta-d)'A(\beta-d)+C$

  3. You should then be able to get it to be a product of two terms and use the theorem you mention

  • 1
    $\begingroup$ This sufficiency proof is an actual consequence of the Pythagorean theorem. $\endgroup$
    – Xi'an
    Aug 26, 2016 at 1:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.