I have the classical regression model

$$y = \beta X + \epsilon$$ $$\epsilon \sim N(0, \sigma^2)$$

where $X$ is taken to be fixed (not random), and $\hat\beta$ is the OLS estimate for $\beta$.

It is known that $(y^T y, X^T y)$ pair is a complete sufficient statistic for $x_0^T \beta$, for some input $x_0$.

Can we conclude that $(y^T y, X^T y)$ is also a sufficient statistic for $\beta$, and why? I think for this to work $X^T X$ should be full rank. I mean a 1 to 1 transformation of a sufficient statistic is still a sufficient statistic, but it is still a sufficient statistic for $x_0^T \beta$. Based on what are we going to conclude the sufficiency of $\hat\beta$ for $\beta$ itself?

  • 2
    Sufficiency of an estimator means that the data is uninformative about $\beta$ given the estimator. If the full rank condition fails here, then $\beta$ is not uniquely identified. This means there is a $\beta^*$ different from $\beta$ such that $\Pr_{\beta^*}(y \leq t) = Pr_{\beta}(y \leq t)$ for all $t$. The data as a whole is not informative about whether the parameter is equal to $\beta$ or $\beta^*$, so it also isn't conditional on some function of it. – CloseToC Nov 2 at 12:26
up vote 7 down vote accepted
+100

Sometimes the simplest way to look at sufficiency is by looking directly at the log-likelihood and using the factorisation theorem. For a linear regression model with Gaussian error term the log-likelihood function can be written as:

$$\begin{equation} \begin{aligned} \ell_{\mathbf{y}, \mathbf{x}}(\boldsymbol{\beta}, \sigma) &= - n \ln \sigma -\frac{1}{2 \sigma^2} || \mathbf{y} - \mathbf{x} \boldsymbol{\beta} ||^2 \\[6pt] &= - n \ln \sigma -\frac{1}{2 \sigma^2} (\mathbf{y} - \mathbf{x} \boldsymbol{\beta})^\text{T} (\mathbf{y} - \mathbf{x} \boldsymbol{\beta} ) \\[6pt] &= - n \ln \sigma -\frac{1}{2 \sigma^2} (\mathbf{y}^\text{T} \mathbf{y} - \mathbf{y}^\text{T} \mathbf{x} \boldsymbol{\beta} - \boldsymbol{\beta}^\text{T} \mathbf{x}^\text{T} \mathbf{y} + \boldsymbol{\beta}^\text{T} \mathbf{x}^\text{T} \mathbf{x} \boldsymbol{\beta} ) \\[6pt] &= - n \ln \sigma -\frac{1}{2 \sigma^2} \mathbf{y}^\text{T} \mathbf{y} -\frac{1}{2 \sigma^2} ( 2 \boldsymbol{\beta}^\text{T} \mathbf{T}_1 - \boldsymbol{\beta}^\text{T} \mathbf{T}_2 \boldsymbol{\beta} ) \\[6pt] &= h(\mathbf{x}, \mathbf{y}, \sigma) + g_\boldsymbol{\beta}(\mathbf{T}_1, \mathbf{T}_2, \sigma), \\[6pt] \end{aligned} \end{equation}$$

where $\mathbf{T}_1 \equiv \mathbf{T}_1(\mathbf{x}, \mathbf{y}) \equiv \mathbf{x}^\text{T} \mathbf{y}$ and $\mathbf{T}_2 \equiv \mathbf{T}_2(\mathbf{x}, \mathbf{y}) \equiv \mathbf{x}^\text{T} \mathbf{x}$. This shows that the statistic $\mathbf{T} \equiv (\mathbf{T}_1, \mathbf{T}_2)$ is sufficient for the coefficient parameter $\boldsymbol{\beta}$. There is no requirement that the design matrix be of full rank for sufficiency, but if it is not of full rank then these statistics are not minimal sufficient (and you obtain a minimal sufficient statistic by reducing the design matrix to full rank).

From the above form we can also see that the OLS estimator $\hat{\boldsymbol{\beta}}$ is not sufficient for $\boldsymbol{\beta}$. Sufficiency also requires knowledge of the matrix $\mathbf{T}_2 = \mathbf{x}^\text{T} \mathbf{x}$, which arises as part of the covariance of the OLS estimator. This tells us that, in the case where the design matrix is of full rank, the OLS estimator and its covariance matrix are jointly sufficient for the unkonwn coefficient parameter.

  • Great explanation. Thank you. – Cowboy Trader Nov 8 at 11:45
  • $(T_1,T_2)$ is a sufficient statistic for $\beta$ according to the above factorization. However I am not sure whether $(\hat\beta,X^TX)$ is sufficient unless covariance matrix is full rank, as we may not recover $X^Ty$ uniquely, no? Or can we rely on the $(X^TX)(X^TX)^+ = I$ penrose property? – Cowboy Trader Nov 8 at 12:21
  • I think the factorization theorem applies to likelihoods, not log-likelihoods. Also, $h$ is free of the parameter, which your notation does not reflect. – Taylor Nov 8 at 19:28
  • @CowboyTrader the last equality sign. But, yes, it's relatively minor. – Taylor Nov 8 at 19:46

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.