I am currently writing some code which performs regression and have noticed that when I calculate variance of $c\hat{\beta}$ I am sometimes on some datasets getting negative values.

The variance is given as $c'(X'X)^{-1}c\frac{e'e}{n - p}$ where $c$ is a vector representing the contrast to be tested, $X$ is the design matrix, $e$ is a vector of residuals and $n$ is the number of datapoints and $p$ is the number of parameters (including an intercept).

I want to compute a T statistic but this involves square rooting the variance and is failing in some cases when the variance is negative. My question is could there be a mathematical/statistical reason why I am having this issue? And if so why/why not? (or should I be asking this instead on StackOverflow?)

My datasets vary in size and negative values appear to occur the most frequently when $n$ is only just greater than $p$ (e.g. when $n=p+1,n=p+2,... $etc ).

  • 1
    $\begingroup$ My guess is that when, $n\approx p$ invertability of matrix becomes numerically unstable due to your choice of how to evaluate the expression for the variance, have you inspected the behavior of your matrix inversions? Any warnings? $\endgroup$ – Jesper Hybel Jan 2 at 18:51
  • 2
    $\begingroup$ My question would be, why are you writing your own code to do this? Nearly every modern language has statistical packages capable of performing regression. $\endgroup$ – StatsStudent Jan 2 at 18:59
  • 1
    $\begingroup$ Are you calculating a generalized inverse? Also, are you rounding your results? $\endgroup$ – StatsStudent Jan 2 at 19:00
  • 1
    $\begingroup$ My bet is that you are not calculating a generalized inverse and if $(X'X)$ your matrix is singular and your program is outputting -1 or some other garbage answer. Are you able to share your code or data? $\endgroup$ – StatsStudent Jan 2 at 19:14
  • 1
    $\begingroup$ @Carl Experience teaches that there are many other ways to compute a negative quantity for a variance. Some variances are computed as differences of two positive values and floating point error is enough to create a negative result in the computer. The comments preceding yours suggest plausible mechanisms for the formula in the question to produce negative values, depending on how the calculation is performed. $\endgroup$ – whuber Jan 2 at 21:17

Your Answer

By clicking "Post Your Answer", you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.