Chi-square test for checking if values are close to zero?

Question

I have a number of set of data points. For each set I have done linear regression to find the lines of best fit for the data.

I hypothesise that the gradients of all the lines of best fit should be zero.

I took the average and standard deviation of all the gradients and these are the values I calculated:

mean = $5.72 \times 10^{-5}$, standard deviation = $5.02 \times 10^{-4}$

To me, a layman, I would say that this shows that the gradients should be zero given the very small mean value and the tight standard distribution.

However I've been told that I need to be a bit more rigorous - apparently I should do a chi-squared test to prove my hypothesis.

Could someone please give me a set of steps detailing how I would go about this?

Thanks.

Answer 1

This problem could be complex but it has a simple solution.

Let me start by explaining why the mean and SD of the estimated slopes are not relevant for testing whether all the true slopes are zero. Consider several illustrative hypothetical alternatives:

1. The true slopes are all zero, so that any nonzero estimates are the result of chance variation in the errors.

2. The true slopes all differ from zero, their estimation errors are small enough to show that every estimated slope significantly differs from zero, but the average slope is close to zero.

3. Some of the true slopes are zero but at least one is not, and for that one the estimated slope is significantly different from zero.

Note, in passing, that changing the units of measurement of the dependent variable will simultaneously change all estimated slopes and the SD of those slopes by the same amount, so that only the ratio mean(slope):SD(slope) is meaningful. In this case that ratio is approximately 1:10. Although this indicates there is a wide variation in estimated slopes, it cannot distinguish among any of these alternatives. That is why these two statistics are useless.

One problem is that these regressions are not necessarily related. They could involve different values of the independent variables. More importantly, the variation of the dependent variables could differ. Imagine (again hypothetically) a series of experiments in which a crude (but fast and cheap) measurement is made of the dependent variable, and then later is followed up by other experiments in which the measurement is made in a more precise way. We couldn't just dump the combined data into one regression model, due to the possibly large differences in distributions of the errors.

Another problem is that the residuals might not have Normal distributions. When each regression includes enough data (typically 30 points is "enough," but as always it depends), the sampling distribution of the estimated slope is still approximately normal, so the t-testing apparatus applies. With small amounts of data, or when some data have high leverage, the t-tests are suspect. However, that's a common problem having many cures, including using generalized linear models, re-expressing (transforming) the independent variables and/or the dependent variable, and other more specialized approaches. So let's assume that the regressions have been appropriately done. This means that we can trust the p-values for the slope tests.

Now we're off and running. The p-value for a single regression, under the null hypothesis that the slope is zero, will have a uniform distribution. Therefore the p-values obtained from each regression should behave like a set of independent draws from a uniform distribution. Small p-values suggest significant differences, so we are interested in whether there are more small p-values than would be expected by chance. (We should also be interested in whether the set of p-values really does look uniform: significant deviations from uniformity would suggest problems with the regressions or subtle violations of the null hypothesis.) A useful statistic, then, is the minimum p-value.

The axioms of probability immediately imply the minimum $p$ of $n$ independent draws from a uniform distribution has a CDF of $\Pr{[p \le t]} = 1-\Pr{[p \gt t]} = 1-(1-t)^n$. To test the null hypothesis that all slopes are zero at the $\alpha$ level (say $\alpha=0.05$, corresponding to $1-0.05 = 95\%$ confidence), we therefore follow this simple procedure:

(a) Let $p_0$ be the smallest p-value of all the $n$ regressions.

(b) Compare $1-(1-p_0)^n$ to $\alpha$. If it is smaller, conclude that at least one slope is nonzero. If it is not smaller, do not reject the hypothesis that all slopes are zero.

Note that this is algebraically the same as checking whether $p_0$ is smaller than $1-(1-\alpha)^{1/n}$. For small $\alpha$ this is close to $\alpha/n$. (The relative error made by this Bonferroni approximation is always less than $-\log(1-\alpha)/\alpha$, which is about $1+\alpha/2$. For $\alpha=0.05$ that's a relative error of about $1.025$, which for most purposes is negligible.) These considerations lead to a simple procedure indeed:

Reject the hypothesis that all slopes are zero only when the smallest p-value of the regressions is less than $\alpha/n$.

Answer 2

"To me, a layman, I would say that this shows that the gradients should be zero given the very small mean value and the tight standard distribution."

One can infer from the numbers you provided (using, e.g., Chebyshev's inequality) that the observed gradients, not only their mean and SD, were small. To the extent that the gradients derived from the data sets are independent observations (from similar distributions) of one underlying "true" gradient this is evidence that the parameter itself is small.

That the SD is ten times larger than the mean is consistent with the idea that the observed gradients are random errors around a true value of zero. It is also consistent with the mean being small (but nonzero) for systematic reasons, maybe similar to whatever reasons led to the suspicion that the mean is exactly zero, and observational noise larger than the true size of the mean.

To say more, additional information is needed about the number, size and nature of the data sets used to determine the gradients.

Answer 3

The answer to your question would depend greatly upon a number of things you didn't mention, but what you're attempting is called a random coefficients analysis (RCA) and you'd want a t-test here against 0. It might be better to just generate a confidence interval around your coeffcient.

One might say that your standard deviation isn't small, it's rather large. It's 10x larger than the slope. Then again, I can't tell if any of the values are large or small because you haven't given a reasonable range of values that it could possibly be. Let's say the predictor in your regression is some very wide ranging value, perhaps mass of ships in grams. If your predicted value was something with a relatively small but meaningful range, like length of time for the hull to rust through in years, then your coefficient would be a very small number but still be a very large effect. So you need to give a lot more information before you can get a good answer.