1
$\begingroup$

How to prove the sufficiency of the least squares estimator?

Let $Y|X=Y_{1},...,Y_{n}$ a sequence of i.i.d random variables conditionated on the values of (fixed) $\mathbf{X}$, where \begin{equation} \mathbf{X}=\left[\begin{array}{cccc} 1 & x_{11} & ... & x_{k1} \\ 1 & x_{12} & ... & x_{k2} \\ \vdots & \vdots & \ddots & \vdots \\ 1 & x_{1n} & ... & x_{kn} \end{array}\right] \end{equation} The least squares estimator is defined by \begin{equation} \boldsymbol{\hat{\beta}}=(\mathbf{X'X})^{-1}\mathbf{X'}\mathbf{Y}\sim \mathcal{N}(\boldsymbol\beta,\boldsymbol\Omega=\sigma^{2}(X'X)^{-1}) \end{equation}

Then the densify function of the estimator is \begin{equation} f_{\hat{\boldsymbol\beta}}(\hat{\boldsymbol\beta}|\boldsymbol\beta,\boldsymbol\Omega)=\frac{1}{(2\pi)^{p/2}|\boldsymbol\Omega|^{1/2}}\exp\left\{-\frac{1}{2}(\boldsymbol{\hat{\beta}}-\boldsymbol\beta)'\boldsymbol\Omega^{-1}(\boldsymbol{\hat{\beta}}-\boldsymbol\beta)\right\} \end{equation}

and the likelihood of the sample is

\begin{align} f_{Y|X,\boldsymbol{\beta}\sigma^{2}I_{k}}(Y|X,\boldsymbol{\beta},\sigma^{2}I_{k})=\left(\frac{1}{2\pi\sigma^{2}}\right)^{n/2}\exp\left\{-\frac{1}{2\sigma^{2}}(Y-X\boldsymbol{\beta})'(Y-X\boldsymbol{\beta})\right\} \end{align}

I know that the factorization theorem states that

Let $X_{1}, X_{2}, ..., X_{n}$ denote random variables with joint probability density function $f(x_{1}, x_{2}, ..., x_{n}; θ)$, which depends on the parameter $θ$.

Then, the statistic $Y=u(X1,X2,...,Xn)$

is sufficient for θ if and only if the p.d.f can be factored into two components, that is:

\begin{equation} f(x_{1},x_{2},...,x_{n};θ)=\phi[u(x_{1},x_{2},...,x_{n});θ]h(x1,x2,...,xn) \end{equation}

How to proceed in this case?

$\endgroup$
  • 1
    $\begingroup$ This deserves a self-study tag, I believe, and you have to detail why you cannot establish the sufficiency on your own. $\endgroup$ – Xi'an Aug 26 '16 at 0:07
  • 1
    $\begingroup$ Okay. I will edit the question. $\endgroup$ – Héctor Aug 26 '16 at 0:09
  • $\begingroup$ What do you know about sufficiency? What have you tried? $\endgroup$ – Glen_b Aug 26 '16 at 0:42
  • $\begingroup$ @Glen_b I have edited the question $\endgroup$ – Héctor Aug 26 '16 at 0:48
3
$\begingroup$

Starting with the joint density (/ likelihood)

  1. try expanding the exponent, collecting terms (in the parameter, so you get terms of the form $\beta'A\beta$ and $B\beta$ for example )

  2. "complete the square", rewriting the terms in the form $(\beta-d)'A(\beta-d)+C$

  3. You should then be able to get it to be a product of two terms and use the theorem you mention

$\endgroup$
  • 1
    $\begingroup$ This sufficiency proof is an actual consequence of the Pythagorean theorem. $\endgroup$ – Xi'an Aug 26 '16 at 1:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.