5
$\begingroup$

Consider the standard simple linear regression model: $$ Y_i = \beta_0 + \beta_1 X_i + \epsilon_i, $$ for $i=1,\dots,n$. In matrix-vector form this is $$ \mathbf{Y} = \mathbf{X_n}\beta + \epsilon, $$ where, in particular, $$ \mathbf{X_n} = \begin{bmatrix} 1 & X_1, \\ 1 & X_2, \\ \vdots & \vdots \\ 1 & X_n \end{bmatrix}. $$

In the case of simple linear regression, the determinant of the matrix $\mathbf{X_n}^T\mathbf{X_n}$ is $$ \text{Det}(\mathbf{X_n}^T\mathbf{X_n}) = n \sum_{i=1}^n(X_i - \overline X_n)^2, $$ where $\overline X_n = \frac{1}{n} \sum_{i=1}^n X_i$.

Now consider multiple regression with $p$ regressors, in which case: $$ \mathbf{X_n} = \begin{bmatrix} 1 & X_{11} & X_{12} & \dots & X_{1p} \\ 1 & X_{21} & X_{22} & \dots & X_{2p} \\ \vdots & \vdots & \vdots & \ddots \vdots \\ 1 & X_{n1} & X_{n2} & \dots & X_{np} \end{bmatrix}. $$

Each rows is an iid observation of the (population) covariates. While each row corresponds to an iid observation of the covariates, the random variables in each row can be dependent on one another.

So for the matrix $\mathbf{X_n}$ in the case of multiple regression:

  1. Is there a known formula for the determinant of $\text{Det}(\mathbf{X_n}^T\mathbf{X_n})$? Of course, the general formula for the determinant of $(p+1)\times (p+1)$ matrix is one such formula, but I am wondering if there is something 'nicer'? The formula for simple linear regression is really nice because it is just a sum over $(X_i - \overline X_n)^2$, is there something analogous for the case of multiple regression? Or at the very least is there something nicer than the general determinant formula?
  2. Is $\text{Det}(\mathbf{X_n}^T\mathbf{X_n})$ guaranteed to be positive like it was in the case of simple linear regression?
  3. Note that in the case of simple linear regression $$\lim_{n \to \infty}\frac{1}{n^{2}}E[\text{Det}(\mathbf{X_n}^T\mathbf{X_n})] = \lim_{n \to \infty}\frac{1}{n} \sum_{i=1}^n(X_i - \overline X_n)^2 = \lim_{n \to \infty}\frac{n-1}{n}\sigma^2 \to \sigma^2.$$ Now, for the multiple regression case, what does $\frac{1}{n^{p+1}}E[\text{Det}(\mathbf{X_n}^T\mathbf{X_n})]$ converge to as $n \to \infty$? Does it convergerge to some population statistic like the simple regression case? Note, we need the scaling to prevent it blowing up, see the comment by jld below.
$\endgroup$
10
  • 1
    $\begingroup$ $det(X^T X)$ is non negative, but will be zero iff there exists a linear dependence among columns of $X$ so not guaranteed to be positive $\endgroup$
    – jcken
    Mar 22 at 18:17
  • $\begingroup$ In (3), what are you assuming about the distribution of the $X_n$? @Erik How are you computing the determinants of non-square matrices?? $\endgroup$
    – whuber
    Mar 22 at 18:37
  • $\begingroup$ Oh yes, good point. $\endgroup$ Mar 22 at 18:37
  • 1
    $\begingroup$ @Erik I wondered, because there are methods to construct determinants from rectangular matrices. I describe one at stats.stackexchange.com/a/512862/919. $\endgroup$
    – whuber
    Mar 22 at 18:39
  • 1
    $\begingroup$ Yes, you do need conditions. The expectation of the determinant depends on the specifics of the process that creates a sequence of rows of $X.$ (My earlier comment referred to a now-deleted comment wherein another user referred to the determinants of $X$ and $X^\prime,$ btw). $\endgroup$
    – whuber
    Mar 22 at 19:10
1
$\begingroup$

Just the 2) since I am not professor: $\mathbf X^T \mathbf X$ is positive definite where $\mathsf {rk}(\mathbf X)=max(n,p+1)$ or maximum number of independent rows which leads the determinant you ask is positive since it is a product of $p+1$ eigenvalues of $\mathbf X^T \mathbf X$.

It's more fair to write $\mathbf X_{n \times {p+1}}$ because this is the shape of the matrix.

PS: Statisticians usually write transposed matrix using the $\mathbf X'$.

$\endgroup$
5
  • $\begingroup$ There no need to write matrix as bold, but since I followed you. $\endgroup$
    – Good Luck
    Mar 22 at 23:07
  • 1
    $\begingroup$ This answer is incorrect: $X^\prime X$ is not guaranteed to be positive definite. One or more of the eigenvalues can be zero. $\endgroup$
    – whuber
    Mar 28 at 16:40
  • $\begingroup$ As I checked the condition $\mathsf {rk}(\mathbf X)= p+1$ is sufficient for the covariance matrix to have all positive eigenvalues. $\endgroup$
    – Good Luck
    Mar 28 at 20:51
  • $\begingroup$ Although that statement is correct, it does not fix what you wrote: "the determinant you ask is positive since it is a product of p+1 eigenvalues of $X^\prime X.$" That is erroneous because it implicitly assumes $X^\prime X$ is positive definite. $\endgroup$
    – whuber
    Mar 28 at 20:54
  • $\begingroup$ Thanks for your attention I will thinker more and perhaps smart update in the next days. $\endgroup$
    – Good Luck
    Mar 28 at 20:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.