0
$\begingroup$

I'm calculating the distribution of the sum of the squares of the components of the MLE $\hat{\beta}$ in linear regression with normal errors. We are assuming that $\beta = 0$. The distribution of the MLE is therefore $\hat{\beta} \sim N(0, \sigma^2 (X^TX)^{-1})$ where $X$ is the design matrix of the model and $\sigma^2$ is the variance of the normally distributed errors.

I want to show that the distribution of $\frac{||\hat{\beta}||^2}{\sigma^2}$ is $\sum_{i=1}^{n-p}\lambda^{-1}W_i$, where the $W_i$ are $\chi_1^2$ random variables, and the $\lambda_i$ are the eigenvalues of $X$ (in this problem we are given that $X$ is full rank and has an eigendecomposition into $X = UDV^T$).

I've already calculated up to $\hat{\beta} \sim N(0, \sigma^2(X^TX)^{-1}) = N(0, VD^{-2}V^T)$ but I don't know how to proceed to find the distribution of $||\hat{\beta}||^2 / \sigma^2$, could anyone help with this?

$\endgroup$

2 Answers 2

1
$\begingroup$

To be accurate, "$X = UDV^T$" is not the "eigendecomposition", but "singular value decomposition (SVD)" of $X$. And with this SVD of $X$, it must be made clear that the size of $D$ is $n \times p$ (assuming $X$ is $n \times p$), and $U$ and $V$ are order $n$ and order $p$ orthogonal matrix respectively. So the notation "$D^{-2}$" does not make sense (unless you are using the "Thin SVD" in which $U$ is $n \times p$ and $D$ is $p \times p$ as opposed to the common "Full SVD" as above). The correct way is expressing $D$ as $D = \begin{bmatrix}\Lambda \\ 0\end{bmatrix}$, where $\Lambda = \operatorname{diag}(\sigma_1, \ldots, \sigma_p)$ is an order $p$ diagonal matrix with $\sigma_1 \geq \cdots \geq \sigma_p > 0$ (which are called singular values of $X$. The relationship between the eigenvalues $\lambda_j$, $1 \leq j \leq p$ of the matrix $X^TX$ and $\sigma_j$ are given by $\lambda_j = \sigma_j^2$). Therefore, the true distribution of $\hat{\beta}$ is \begin{align} \hat{\beta} \sim N_p(0, \sigma^2V\Lambda^{-2}V^T), \end{align} which implies that $\sigma^{-1}\hat{\beta} \overset{d}{=} A\xi$, where $\xi \sim N_p(0, I_{(p)})$ and $A$ satisfies $A^2 = V\Lambda^{-2}V^T$. In addition, since $V$ is an order $p$ orthogonal matrix, $V^T\xi := W \sim N_p(0, I_{(p)})$. It then follows that \begin{align} \sigma^{-2}\|\hat{\beta}\|^2 \overset{d}{=} \xi^TA^2\xi = \xi^TV\Lambda^{-2}V^T\xi = W^T\Lambda^{-2}W = \sum_{j = 1}^p\sigma_j^{-2}W_j^2 = \sum_{j = 1}^p\lambda_j^{-1}W_j^2. \end{align}

Therefore, the result you presented is incorrect in that:

  1. There are $p$ summands, not $n - p$ summands. To see this more clearly, just consider the simplest $X = \begin{bmatrix}I_{(p)} \\ 0\end{bmatrix}$, for which case $\hat{\beta} \sim N_p(0, \sigma^2I_{(p)})$.
  2. $\lambda_j$ is the eigenvalue of $X^TX$, not of $X$. In fact, as a rectangular matrix, it is impossible for $X$ to have eigenvalues.
$\endgroup$
4
  • $\begingroup$ (+1) But the complaints about notation are distracting. There is a standard formulation of SVD in which "$D^{-2}$" is often used to refer to the $p\times p$ (not $n\times n$) matrix whose nonzero diagonal elements are $d_i^{-2},$ where $d_i$ are the nonzero diagonal elements of $D.$ Although that needs clarifying, it could be done a little less critically. It might also be worth pointing out that you used the full-rank assumption about $X$ when you concluded there are $p$ terms in the sum. $\endgroup$
    – whuber
    Commented May 22, 2023 at 13:44
  • 1
    $\begingroup$ @whuber I am not really "criticizing/distracting" OP. IMO, I have to clarify all the dimensions and basic concepts first before answering this question correctly. For the dimensionality of $D$ (either $n \times p$ in my answer or $p \times p$ as you suggested), I believe I follow the widely-accepted SVD formulation (en.wikipedia.org/wiki/Singular_value_decomposition). I do have seen some author made $D$ to be $p \times p$, but in that case the dimension of $U$ needs to be adapted accordingly. But I see your point and will edit the wording a bit. $\endgroup$
    – Zhanxiong
    Commented May 22, 2023 at 14:00
  • $\begingroup$ As for the full-rank assumption, it was clearly mentioned in OP's question. $\endgroup$
    – Zhanxiong
    Commented May 22, 2023 at 14:04
  • $\begingroup$ Right: my point was only that it could be helpful to point out how you used the full-rank assumption in this solution. My comment about being critical originates from a general sense of negativity in the post, beginning with stating what $X=UDV^\prime$ is not and culminating with a summary of what is incorrect. The same points can be made in the same (mathematical and statistical) ways from a positive perspective. Just start off by saying what the SVD really is and end by summarizing what the correct result is. IMHO, such a positive approach is less likely to be misread. $\endgroup$
    – whuber
    Commented May 22, 2023 at 16:49
0
$\begingroup$

If you know the mean μ=0 and variance (let’s call it Σ) of β, then you can compute the expected value and variance at least.

The expected value is tr(Σ).

The variance is 2 tr(Σ ⋅ Σ).

https://en.wikipedia.org/wiki/Quadratic_form_(statistics)?wprov=sfti1

This distribution is a Generalized Chi ² distribution https://en.wikipedia.org/wiki/Generalized_chi-squared_distribution?wprov=sfti1

It may be worth considering if you really want the sum of squares ( β’ β ) , or if you’d be just as happy with β’ Σ⁻¹ β. That is a lot easier. It follows a standard Chi² distribution I think.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.