I am currently writing some code which performs regression and have noticed that when I calculate variance of $c\hat{\beta}$ I am sometimes on some datasets getting negative values.

The variance is given as $c'(X'X)^{-1}c\frac{e'e}{n - p}$ where $c$ is a vector representing the contrast to be tested, $X$ is the design matrix, $e$ is a vector of residuals and $n$ is the number of datapoints and $p$ is the number of parameters (including an intercept).

I want to compute a T statistic but this involves square rooting the variance and is failing in some cases when the variance is negative. My question is could there be a mathematical/statistical reason why I am having this issue? And if so why/why not? (or should I be asking this instead on StackOverflow?)

My datasets vary in size and negative values appear to occur the most frequently when $n$ is only just greater than $p$ (e.g. when $n=p+1,n=p+2,... $etc ).

  • 1
    $\begingroup$ My guess is that when, $n\approx p$ invertability of matrix becomes numerically unstable due to your choice of how to evaluate the expression for the variance, have you inspected the behavior of your matrix inversions? Any warnings? $\endgroup$ – Jesper Hybel Jan 2 at 18:51
  • 2
    $\begingroup$ My question would be, why are you writing your own code to do this? Nearly every modern language has statistical packages capable of performing regression. $\endgroup$ – StatsStudent Jan 2 at 18:59
  • 1
    $\begingroup$ Are you calculating a generalized inverse? Also, are you rounding your results? $\endgroup$ – StatsStudent Jan 2 at 19:00
  • 1
    $\begingroup$ My bet is that you are not calculating a generalized inverse and if $(X'X)$ your matrix is singular and your program is outputting -1 or some other garbage answer. Are you able to share your code or data? $\endgroup$ – StatsStudent Jan 2 at 19:14
  • 1
    $\begingroup$ @Carl Experience teaches that there are many other ways to compute a negative quantity for a variance. Some variances are computed as differences of two positive values and floating point error is enough to create a negative result in the computer. The comments preceding yours suggest plausible mechanisms for the formula in the question to produce negative values, depending on how the calculation is performed. $\endgroup$ – whuber Jan 2 at 21:17

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.