7
$\begingroup$

I have the classical regression model

$$y = \beta X + \epsilon$$ $$\epsilon \sim N(0, \sigma^2)$$

where $X$ is taken to be fixed (not random), and $\hat\beta$ is the OLS estimate for $\beta$.

It is known that $(y^T y, X^T y)$ pair is a complete sufficient statistic for $x_0^T \beta$, for some input $x_0$.

Can we conclude that $(y^T y, X^T y)$ is also a sufficient statistic for $\beta$, and why? I think for this to work $X^T X$ should be full rank. I mean a 1 to 1 transformation of a sufficient statistic is still a sufficient statistic, but it is still a sufficient statistic for $x_0^T \beta$. Based on what are we going to conclude the sufficiency of $\hat\beta$ for $\beta$ itself?

$\endgroup$
1
  • 3
    $\begingroup$ Sufficiency of an estimator means that the data is uninformative about $\beta$ given the estimator. If the full rank condition fails here, then $\beta$ is not uniquely identified. This means there is a $\beta^*$ different from $\beta$ such that $\Pr_{\beta^*}(y \leq t) = Pr_{\beta}(y \leq t)$ for all $t$. The data as a whole is not informative about whether the parameter is equal to $\beta$ or $\beta^*$, so it also isn't conditional on some function of it. $\endgroup$
    – CloseToC
    Commented Nov 2, 2018 at 12:26

2 Answers 2

16
+100
$\begingroup$

Sometimes the simplest way to look at sufficiency is by looking directly at the log-likelihood and using the factorisation theorem. For a linear regression model with Gaussian error term the log-likelihood function can be written as:

$$\begin{equation} \begin{aligned} \ell_{\mathbf{y}, \mathbf{x}}(\boldsymbol{\beta}, \sigma) &= - n \ln \sigma -\frac{1}{2 \sigma^2} || \mathbf{y} - \mathbf{x} \boldsymbol{\beta} ||^2 \\[6pt] &= - n \ln \sigma -\frac{1}{2 \sigma^2} (\mathbf{y} - \mathbf{x} \boldsymbol{\beta})^\text{T} (\mathbf{y} - \mathbf{x} \boldsymbol{\beta} ) \\[6pt] &= - n \ln \sigma -\frac{1}{2 \sigma^2} (\mathbf{y}^\text{T} \mathbf{y} - \mathbf{y}^\text{T} \mathbf{x} \boldsymbol{\beta} - \boldsymbol{\beta}^\text{T} \mathbf{x}^\text{T} \mathbf{y} + \boldsymbol{\beta}^\text{T} \mathbf{x}^\text{T} \mathbf{x} \boldsymbol{\beta} ) \\[6pt] &= - n \ln \sigma -\frac{1}{2 \sigma^2} \mathbf{y}^\text{T} \mathbf{y} -\frac{1}{2 \sigma^2} ( 2 \boldsymbol{\beta}^\text{T} \mathbf{T}_1 - \boldsymbol{\beta}^\text{T} \mathbf{T}_2 \boldsymbol{\beta} ) \\[6pt] &= h(\mathbf{y}, \sigma) + g_\boldsymbol{\beta}(\mathbf{T}_1, \mathbf{T}_2, \sigma), \\[6pt] \end{aligned} \end{equation}$$

where $\mathbf{T}_1 \equiv \mathbf{T}_1(\mathbf{x}, \mathbf{y}) \equiv \mathbf{x}^\text{T} \mathbf{y}$ and $\mathbf{T}_2 \equiv \mathbf{T}_2(\mathbf{x}, \mathbf{y}) \equiv \mathbf{x}^\text{T} \mathbf{x}$. This shows that the statistic $\mathbf{T} \equiv (\mathbf{T}_1, \mathbf{T}_2)$ is sufficient for the coefficient parameter $\boldsymbol{\beta}$. There is no requirement that the design matrix be of full rank for sufficiency, but if it is not of full rank then these statistics are not minimal sufficient (and you obtain a minimal sufficient statistic by reducing the design matrix to full rank).

From the above form we can also see that the OLS estimator $\hat{\boldsymbol{\beta}}$ is not sufficient for $\boldsymbol{\beta}$. Sufficiency also requires knowledge of the matrix $\mathbf{T}_2 = \mathbf{x}^\text{T} \mathbf{x}$, which arises as part of the covariance of the OLS estimator. This tells us that, in the case where the design matrix is of full rank, the OLS estimator and its covariance matrix are jointly sufficient for the unknown coefficient parameter. (Of course, it is worth noting that regression problems always condition on $\mathbf{x}$, so in this context we get the required sufficiency.)

$\endgroup$
5
  • $\begingroup$ $(T_1,T_2)$ is a sufficient statistic for $\beta$ according to the above factorization. However I am not sure whether $(\hat\beta,X^TX)$ is sufficient unless covariance matrix is full rank, as we may not recover $X^Ty$ uniquely, no? Or can we rely on the $(X^TX)(X^TX)^+ = I$ penrose property? $\endgroup$ Commented Nov 8, 2018 at 12:21
  • $\begingroup$ I think the factorization theorem applies to likelihoods, not log-likelihoods. Also, $h$ is free of the parameter, which your notation does not reflect. $\endgroup$
    – Taylor
    Commented Nov 8, 2018 at 19:28
  • $\begingroup$ @CowboyTrader the last equality sign. But, yes, it's relatively minor. $\endgroup$
    – Taylor
    Commented Nov 8, 2018 at 19:46
  • $\begingroup$ "There is no requirement that the design matrix be of full rank for sufficiency", "in the case where the design matrix is of full rank, the OLS estimator and its covariance matrix are jointly sufficient". I find these two statements somewhat contradicting. So is full rank a requirement for sufficiency or just minimal sufficiency? $\endgroup$ Commented Jan 10, 2020 at 10:08
  • $\begingroup$ It is only a requirement for minimal sufficiency. $\endgroup$
    – Ben
    Commented Jan 10, 2020 at 10:54
0
$\begingroup$

I was looking for some detailed proof for the simple linear model. By the factorisation theorem we have the following solution

Considering the classical model

$$y_i=\beta_0+\beta_1 x_i+\varepsilon_i,\ \mbox{ em que}\ \varepsilon_i\sim \mathrm{N}(0;\sigma^2),\ \hbox{Cov}(\varepsilon_i,\varepsilon_j)=0, i\neq j,\ i,j=1,\ldots, n$$

We start from $\mathbf{y}=(y_1,y_2,\ldots,y_n)$ a simple random sample and $x_1,x_2,\ldots, x_n$ not random. $\beta_0$ and $\beta_1$ are the classical parameters for the model (both maximum likehood or OLS). Consequently

$$y_i \sim \mathrm{N}(\beta_0+\beta_1 x_i;\sigma^2),\ i=1,\ldots, n.$$

Density function for $y_i$ $$f(y_i)= \frac{1}{\sigma \sqrt{2\pi}} \exp\left\{ -\frac{1}{2\sigma^2}\left[y_i-(\beta_0+\beta_1 x_i)\right]^2 \right\}\ -\infty <x_i<\infty $$

Likehood function

\begin{align*} \hbox{L}(\mathbf{y}) &= \prod_{i=1}^{n} \frac{1}{\sigma \sqrt{2\pi}} \exp\left\{ -\frac{1}{2\sigma^2}\left[(y_i-\overline{y})-(\beta_0+\beta_1 x_i-\overline{y})\right]^2 \right\},\ -\infty <x_i<\infty \\ &= (2\pi\sigma^2)^{-n/2}\\ &= \exp\left\{ -\frac{1}{2\sigma^2}\left[\underbrace{\sum_{i=1}^{n}(y_i-\overline{y})^2-2\sum_{i=1}^{n}(y_i-\overline{y})(\beta_0+\beta_1 x_i-\overline{y})+\sum_{i=1}^{n}(\beta_0+\beta_1 x_i-\overline{y})^2}_{\delta}\right] \right\} \end{align*}

Be $s_{yy}=\sum_{i=1}^{n}(y_i-\overline{y})^2=\sum_{i=1}^{n}y_i^2-n\overline{y}$ e $s_{xx}=\sum_{i=1}^{n}x_i^2-n\overline{x}$, $y$ random variable e $x$ constant. \begin{align*} \delta&=\sum_{i=1}^{n}(y_i-\overline{y})^2-2\sum_{i=1}^{n}(y_i-\overline{y})(\beta_0+\beta_1 x_i-\overline{y})+\sum_{i=1}^{n}(\beta_0+\beta_1 x_i-\overline{y})^2\\ \delta&=s_{yy}-2\sum_{i=1}^{n}(y_i\beta_0+\beta_1 x_iy_i-\overline{y}^2) +\sum_{i=1}^{n}\left[(\beta_0+\beta_1 x_i)^2-2(\beta_0+\beta_1 x_i)\overline{y}+\overline{y}^2\right]\\ \delta&=s_{yy}-2\alpha\sum_{i=1}^{n}y_i-2\beta\sum_{i=1}^{n}x_iy_i +2n\overline{y}^2+n\beta_0^2+2\beta_0\beta_1\sum_{i=1}^{n}x_i+\beta^2_1\sum_{i=1}^{n}x_i^2-2n\beta_0\overline{y}-2\beta_1\overline{y}\sum_{i=1}^{n}x_i +n\overline{y}^2\\ \delta&=\sum_{i=1}^{n}y_i^2-4n\beta_0\overline{y}+n\beta_0^2-2\beta_1\sum_{i=1}^{n}x_iy_i+2n\overline{y}^2+2\beta_0\beta_1\sum_{i=1}^{n}x_i+\beta^2_1\sum_{i=1}^{n}x_i^2-2n\beta_1\overline{x}\ \overline{y}\\ \delta&=\sum_{i=1}^{n}y_i^2+n\beta_0^2-4n\beta_0\overline{y}+2\beta_0\beta_1\sum_{i=1}^{n}x_i-2\beta_1\sum_{i=1}^{n}x_iy_i+2n\overline{y}^2+\beta^2_1\sum_{i=1}^{n}x_i^2-2n\beta_1\overline{x}\ \overline{y}\\ \delta&=\sum_{i=1}^{n}y_i^2-2n\beta_0 -2n\beta_0\left(\overline{y}-\beta_1\overline{x}\right)-2\beta_1\left(\sum_{i=1}^{n}x_iy_i-n\overline{x}\ \overline{y}\right)+2n\overline{y}^2+\beta^2_1\sum_{i=1}^{n}x_i^2\\ \delta&=\sum_{i=1}^{n}y_i^2+2n\overline{y}^2-2n\beta_0 -2n\beta_0\left(\overline{y}-\beta_1\overline{x}\right)+\beta^2_1\sum_{i=1}^{n}x_i^2-2\beta_1\left(\sum_{i=1}^{n}x_iy_i-n\overline{x}\ \overline{y}\right)\\ \delta &= \underbrace{\sum_{i=1}^{n}y_i^2+2n\overline{y}^2}_{g(\mathbf{y})}-\underbrace{2n\beta_0 -2n\beta_0\left(\overline{y}-\beta_1\overline{x}\right)}_{h(\mathbf{y},\beta_0)}+\underbrace{\beta^2_1 \sum_{i=1}^{n}x_i^2-2 s_{xx}\beta_1 \left(\frac{\sum_{i=1}^{n}x_iy_i-n\overline{x}\ \overline{y}}{s_{xx}}\right)}_{s(\mathbf{y},\beta_1)} \end{align*} Since $\hat{\beta_1}=\frac{\sum_{i=1}^{n}x_iy_i-n\overline{x}\ \overline{y}}{s_{xx}}$ and $\hat{\beta}_0=\overline{y}-\hat{\beta}\overline{x}$ are OLS and ML estimators \begin{eqnarray*} \hbox{L}(y_i) &=&(2\pi\sigma^2)^{-n/2} \exp\left[g(\mathbf{y})\right]\times \exp\left[h(\mathbf{y},\hat{\beta_0})+s(\mathbf{y},\hat{\beta_1})\right] \end{eqnarray*}

Finally by the factorisation theorem we prove that for a simple linear regression model: $$y_i=\beta_0+\beta_1x_i+\varepsilon_i,$$

the conditional distribution

$$Y|\hat{\beta}_0,\hat{\beta}_1$$

do not depends on $\beta_0$ and $\beta_1$, the original parameters for $Y$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.