As I understand it, $\text{standard error} = \frac{\text{standard deviation}}{\sqrt{n}}$, where $n$ is the number of observations. I also know that $cov(\hat{\beta}) = \sigma^{2}(X^{T}X)^{-1}$; this is a pxp variance-covariance matrix. So why is the standard error of the components of the coefficient estimate vector $\boldsymbol{\hat{\beta}}$ in, for example, a normal linear model, $\sigma^{2}[(X^{T}X)^{-1}]_{ii}$ and not $\frac{1}{\sqrt{n}}\sigma^{2}[(X^{T}X)^{-1}]_{ii}$ where the subscript $ii$ denotes the $i,i$th entry of the matrix?

  • $\begingroup$ i asked a related question yesterday, using R... stats.stackexchange.com/questions/262160/… $\endgroup$ – Haitao Du Feb 16 '17 at 21:28
  • 1
    $\begingroup$ The "$n$" factor is implicit already: consider the case where $X$ has been whitened, so $X^TX=nI$. $\endgroup$ – GeoMatt22 Feb 16 '17 at 21:35
  • $\begingroup$ @GeoMatt22 Hmm, yes, I can see when it is whitened it works but I can't generalise it - could you elaborate on where it is implicit? $\endgroup$ – python_learner Feb 16 '17 at 21:57
  • 1
    $\begingroup$ It's just dot products vs. covariances (sums vs. averages): $\boldsymbol{x}^T\boldsymbol{y} = \overline{(xy)}\times{n}$ $\endgroup$ – GeoMatt22 Feb 16 '17 at 22:21
  • 1
    $\begingroup$ @GeoMatt22 I'm afraid I don't find it any easier to understand using the equality you presented. :l Could you provide working? $\endgroup$ – python_learner Feb 16 '17 at 22:54

The "$n$" factor is implicit already in $\boldsymbol{X}^T\boldsymbol{X}$.

This is easiest to see in the case where $\boldsymbol{X}$ has been whitened, so $\boldsymbol{X}^T\boldsymbol{X}=n\boldsymbol{I}$.

More generally we have $\boldsymbol{X}=[\boldsymbol{x}_1,\ldots\boldsymbol{x}_p]$, where $\boldsymbol{x}_p\in\mathbb{R}^n$. Then the entries of the matrix are $$\left(\boldsymbol{X}^T\boldsymbol{X}\right)_{ij}=\boldsymbol{x}_i^T\boldsymbol{x}_j$$ Now choose a pair of columns, $\boldsymbol{a}=\boldsymbol{x}_i$ and $\boldsymbol{b}=\boldsymbol{x}_j$. Then we have $$\boldsymbol{a}^T\boldsymbol{b}=\sum_{k=1}^na_kb_k=\overline{(ab)}\,n$$ So each entry of the matrix is a dot product, which can be seen as $n$ times an average.

In the case where $\boldsymbol{X}$ has been centered so the columns have zero means, the product-matrix entries are covariances.

For an overview of the general case, see here.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.