# Sufficient Statistic for $\beta$ in OLS

I have the classical regression model

$$y = \beta X + \epsilon$$ $$\epsilon \sim N(0, \sigma^2)$$

where $$X$$ is taken to be fixed (not random), and $$\hat\beta$$ is the OLS estimate for $$\beta$$.

It is known that $$(y^T y, X^T y)$$ pair is a complete sufficient statistic for $$x_0^T \beta$$, for some input $$x_0$$.

Can we conclude that $$(y^T y, X^T y)$$ is also a sufficient statistic for $$\beta$$, and why? I think for this to work $$X^T X$$ should be full rank. I mean a 1 to 1 transformation of a sufficient statistic is still a sufficient statistic, but it is still a sufficient statistic for $$x_0^T \beta$$. Based on what are we going to conclude the sufficiency of $$\hat\beta$$ for $$\beta$$ itself?

• Sufficiency of an estimator means that the data is uninformative about $\beta$ given the estimator. If the full rank condition fails here, then $\beta$ is not uniquely identified. This means there is a $\beta^*$ different from $\beta$ such that $\Pr_{\beta^*}(y \leq t) = Pr_{\beta}(y \leq t)$ for all $t$. The data as a whole is not informative about whether the parameter is equal to $\beta$ or $\beta^*$, so it also isn't conditional on some function of it. – CloseToC Nov 2 '18 at 12:26

Sometimes the simplest way to look at sufficiency is by looking directly at the log-likelihood and using the factorisation theorem. For a linear regression model with Gaussian error term the log-likelihood function can be written as:

\begin{aligned} \ell_{\mathbf{y}, \mathbf{x}}(\boldsymbol{\beta}, \sigma) &= - n \ln \sigma -\frac{1}{2 \sigma^2} || \mathbf{y} - \mathbf{x} \boldsymbol{\beta} ||^2 \\[6pt] &= - n \ln \sigma -\frac{1}{2 \sigma^2} (\mathbf{y} - \mathbf{x} \boldsymbol{\beta})^\text{T} (\mathbf{y} - \mathbf{x} \boldsymbol{\beta} ) \\[6pt] &= - n \ln \sigma -\frac{1}{2 \sigma^2} (\mathbf{y}^\text{T} \mathbf{y} - \mathbf{y}^\text{T} \mathbf{x} \boldsymbol{\beta} - \boldsymbol{\beta}^\text{T} \mathbf{x}^\text{T} \mathbf{y} + \boldsymbol{\beta}^\text{T} \mathbf{x}^\text{T} \mathbf{x} \boldsymbol{\beta} ) \\[6pt] &= - n \ln \sigma -\frac{1}{2 \sigma^2} \mathbf{y}^\text{T} \mathbf{y} -\frac{1}{2 \sigma^2} ( 2 \boldsymbol{\beta}^\text{T} \mathbf{T}_1 - \boldsymbol{\beta}^\text{T} \mathbf{T}_2 \boldsymbol{\beta} ) \\[6pt] &= h(\mathbf{x}, \mathbf{y}, \sigma) + g_\boldsymbol{\beta}(\mathbf{T}_1, \mathbf{T}_2, \sigma), \\[6pt] \end{aligned}

where $$\mathbf{T}_1 \equiv \mathbf{T}_1(\mathbf{x}, \mathbf{y}) \equiv \mathbf{x}^\text{T} \mathbf{y}$$ and $$\mathbf{T}_2 \equiv \mathbf{T}_2(\mathbf{x}, \mathbf{y}) \equiv \mathbf{x}^\text{T} \mathbf{x}$$. This shows that the statistic $$\mathbf{T} \equiv (\mathbf{T}_1, \mathbf{T}_2)$$ is sufficient for the coefficient parameter $$\boldsymbol{\beta}$$. There is no requirement that the design matrix be of full rank for sufficiency, but if it is not of full rank then these statistics are not minimal sufficient (and you obtain a minimal sufficient statistic by reducing the design matrix to full rank).

From the above form we can also see that the OLS estimator $$\hat{\boldsymbol{\beta}}$$ is not sufficient for $$\boldsymbol{\beta}$$. Sufficiency also requires knowledge of the matrix $$\mathbf{T}_2 = \mathbf{x}^\text{T} \mathbf{x}$$, which arises as part of the covariance of the OLS estimator. This tells us that, in the case where the design matrix is of full rank, the OLS estimator and its covariance matrix are jointly sufficient for the unkonwn coefficient parameter.

• $(T_1,T_2)$ is a sufficient statistic for $\beta$ according to the above factorization. However I am not sure whether $(\hat\beta,X^TX)$ is sufficient unless covariance matrix is full rank, as we may not recover $X^Ty$ uniquely, no? Or can we rely on the $(X^TX)(X^TX)^+ = I$ penrose property? – Cowboy Trader Nov 8 '18 at 12:21
• I think the factorization theorem applies to likelihoods, not log-likelihoods. Also, $h$ is free of the parameter, which your notation does not reflect. – Taylor Nov 8 '18 at 19:28
• @CowboyTrader the last equality sign. But, yes, it's relatively minor. – Taylor Nov 8 '18 at 19:46

I was looking for some detailed proof for the simple linear model. By the factorisation theorem we have the following solution

Considering the classical model

$$y_i=\beta_0+\beta_1 x_i+\varepsilon_i,\ \mbox{ em que}\ \varepsilon_i\sim \mathrm{N}(0;\sigma^2),\ \hbox{Cov}(\varepsilon_i,\varepsilon_j)=0, i\neq j,\ i,j=1,\ldots, n$$

We start from $$\mathbf{y}=(y_1,y_2,\ldots,y_n)$$ a simple random sample and $$x_1,x_2,\ldots, x_n$$ not random. $$\beta_0$$ and $$\beta_1$$ are the classical parameters for the model (both maximum likehood or OLS). Consequently

$$y_i \sim \mathrm{N}(\beta_0+\beta_1 x_i;\sigma^2),\ i=1,\ldots, n.$$

Density function for $$y_i$$ $$f(y_i)= \frac{1}{\sigma \sqrt{2\pi}} \exp\left\{ -\frac{1}{2\sigma^2}\left[y_i-(\beta_0+\beta_1 x_i)\right]^2 \right\}\ -\infty

Likehood function

\begin{align*} \hbox{L}(\mathbf{y}) &= \prod_{i=1}^{n} \frac{1}{\sigma \sqrt{2\pi}} \exp\left\{ -\frac{1}{2\sigma^2}\left[(y_i-\overline{y})-(\beta_0+\beta_1 x_i-\overline{y})\right]^2 \right\},\ -\infty

Be $$s_{yy}=\sum_{i=1}^{n}(y_i-\overline{y})^2=\sum_{i=1}^{n}y_i^2-n\overline{y}$$ e $$s_{xx}=\sum_{i=1}^{n}x_i^2-n\overline{x}$$, $$y$$ random variable e $$x$$ constant. \begin{align*} \delta&=\sum_{i=1}^{n}(y_i-\overline{y})^2-2\sum_{i=1}^{n}(y_i-\overline{y})(\beta_0+\beta_1 x_i-\overline{y})+\sum_{i=1}^{n}(\beta_0+\beta_1 x_i-\overline{y})^2\\ \delta&=s_{yy}-2\sum_{i=1}^{n}(y_i\beta_0+\beta_1 x_iy_i-\overline{y}^2) +\sum_{i=1}^{n}\left[(\beta_0+\beta_1 x_i)^2-2(\beta_0+\beta_1 x_i)\overline{y}+\overline{y}^2\right]\\ \delta&=s_{yy}-2\alpha\sum_{i=1}^{n}y_i-2\beta\sum_{i=1}^{n}x_iy_i +2n\overline{y}^2+n\beta_0^2+2\beta_0\beta_1\sum_{i=1}^{n}x_i+\beta^2_1\sum_{i=1}^{n}x_i^2-2n\beta_0\overline{y}-2\beta_1\overline{y}\sum_{i=1}^{n}x_i +n\overline{y}^2\\ \delta&=\sum_{i=1}^{n}y_i^2-4n\beta_0\overline{y}+n\beta_0^2-2\beta_1\sum_{i=1}^{n}x_iy_i+2n\overline{y}^2+2\beta_0\beta_1\sum_{i=1}^{n}x_i+\beta^2_1\sum_{i=1}^{n}x_i^2-2n\beta_1\overline{x}\ \overline{y}\\ \delta&=\sum_{i=1}^{n}y_i^2+n\beta_0^2-4n\beta_0\overline{y}+2\beta_0\beta_1\sum_{i=1}^{n}x_i-2\beta_1\sum_{i=1}^{n}x_iy_i+2n\overline{y}^2+\beta^2_1\sum_{i=1}^{n}x_i^2-2n\beta_1\overline{x}\ \overline{y}\\ \delta&=\sum_{i=1}^{n}y_i^2-2n\beta_0 -2n\beta_0\left(\overline{y}-\beta_1\overline{x}\right)-2\beta_1\left(\sum_{i=1}^{n}x_iy_i-n\overline{x}\ \overline{y}\right)+2n\overline{y}^2+\beta^2_1\sum_{i=1}^{n}x_i^2\\ \delta&=\sum_{i=1}^{n}y_i^2+2n\overline{y}^2-2n\beta_0 -2n\beta_0\left(\overline{y}-\beta_1\overline{x}\right)+\beta^2_1\sum_{i=1}^{n}x_i^2-2\beta_1\left(\sum_{i=1}^{n}x_iy_i-n\overline{x}\ \overline{y}\right)\\ \delta &= \underbrace{\sum_{i=1}^{n}y_i^2+2n\overline{y}^2}_{g(\mathbf{y})}-\underbrace{2n\beta_0 -2n\beta_0\left(\overline{y}-\beta_1\overline{x}\right)}_{h(\mathbf{y},\beta_0)}+\underbrace{\beta^2_1 \sum_{i=1}^{n}x_i^2-2 s_{xx}\beta_1 \left(\frac{\sum_{i=1}^{n}x_iy_i-n\overline{x}\ \overline{y}}{s_{xx}}\right)}_{s(\mathbf{y},\beta_1)} \end{align*} Since $$\hat{\beta_1}=\frac{\sum_{i=1}^{n}x_iy_i-n\overline{x}\ \overline{y}}{s_{xx}}$$ and $$\hat{\beta}_0=\overline{y}-\hat{\beta}\overline{x}$$ are OLS and ML estimators $$\begin{eqnarray*} \hbox{L}(y_i) &=&(2\pi\sigma^2)^{-n/2} \exp\left[g(\mathbf{y})\right]\times \exp\left[h(\mathbf{y},\hat{\beta_0})+s(\mathbf{y},\hat{\beta_1})\right] \end{eqnarray*}$$

Finally by the factorisation theorem we prove that for a simple linear regression model: $$y_i=\beta_0+\beta_1x_i+\varepsilon_i,$$

the conditional distribution

$$Y|\hat{\beta}_0,\hat{\beta}_1$$

do not depends on $$\beta_0$$ and $$\beta_1$$, the original parameters for $$Y$$.