6
$\begingroup$

My question is about the distribution of the t-statistics in Weighted Least Squares regression.

I'm finding that for a fixed Y and random X and W, the "t value" (t-statistic) reported by R (and separately calculated by hand in matlab) has an absolute value greater than 2 close to 10% of the time (as opposed to ~5%, which would be expected for random data). Here is my code:

N = 100
runs = 10000
Y = rnorm(N);

tstats = rbind(lapply(seq(10), function(X){W=abs(rnorm(100,1)); X = rnorm(100,1);
                                  summary(lm(Y~X,weights=W))$coefficient[2,3] }))

This code is just running through a loop where for each row of M (which is a 10,000 x 100 random matrix) --- call the row in a given iteration X ---, it gets a new set of random (non-negative) “weights”, and then regresses Y against X using the weights W, and pulls out the "t value" (which is element (2,3) of the coefficients). Since the W’s are random each time, the average correlation between Y and W should be 0 (if that possibly had an affect).

What's worse, even if I skip the weights, the following regression seems to have |t-statistic| greater than 2 about 10% of the time as well:

lm( I(Y*sqrt(W)) ~ I(X*sqrt(W)) )

This regression would have the same inputs, except I'm keeping the intercept (constant vector) and not multiplying it by sqrt(W) as in the weighted regression case. This especially perplexes me.

Alternatively, if you use the code above but replace the weighted lm() there with

lm( Y ~ X )

You will see that the coefficient is significant only 5% of the time.

What is going on here?

$\endgroup$
1
  • $\begingroup$ I changed the original code to make it more clear that X is not fixed. $\endgroup$
    – arithmetic
    Jun 12, 2013 at 16:40

1 Answer 1

8
$\begingroup$

I think the problem is that you are generating the weights at random, uncorrelated with the y value. In a real weighted regression the points with lower variance will have higher weights. Since the true relationship is mean and variance of 0 that means that points furthest from 0 would be consistent with higher variances and therefore lower weights, but you don't given them lower weights, they get random weights which could be high or low giving some more extreme values than expected.

If you do the simulation more realistically by generating a set of weights, then generate Y with variances based on the weights, then analyze (you could use the same set of x's, or randomly generate the x's as well), I would expect the t-values to behave more properly.

Here is a quick example:

tstats <- replicate(1000, { x <- rnorm(N); w <- abs(rnorm(100,1)); 
    y <- rnorm(100, 0, sqrt(1/w));
    coef(summary(lm(y~x, weights=w)))[2,3]})
mean(abs(tstats)>2)

I saw just under 5% as expected.

$\endgroup$
2
  • 1
    $\begingroup$ +1 Good catch. There is nothing in the theory of regression that says when you randomize the weights that the p-value should average 5%! In fact, because the data remain the same during this simulation, we would expect the average p-value (or correspondingly some average t-value) to reflect whatever significance inheres in the data, regardless of the weights. In short, this simulation seems to be just an enormously inefficient way of doing unweighted linear regression. $\endgroup$
    – whuber
    Jun 12, 2013 at 12:05
  • $\begingroup$ Thanks so much for your response. The situation I found myself in was that someone was doing a weighted regression with weights chosen in a way expected (by the theory) to be inversely related to the variance of the data points. However, I found that when I regressed random vectors (X) against the data (Y) using their weights, I got t-statistics with abs value > 2 about 10% of the time. Though they were happily advertising p values of ~0.08. All other data-mining issues aside, it is reasonable to conclude from this brief description, that their choice of weights is affecting the regression? $\endgroup$
    – arithmetic
    Jun 12, 2013 at 16:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.