# Chi-square test for checking if values are close to zero?

I have a number of set of data points. For each set I have done linear regression to find the lines of best fit for the data.

I hypothesise that the gradients of all the lines of best fit should be zero.

I took the average and standard deviation of all the gradients and these are the values I calculated:

mean = $5.72 \times 10^{-5}$, standard deviation = $5.02 \times 10^{-4}$

To me, a layman, I would say that this shows that the gradients should be zero given the very small mean value and the tight standard distribution.

However I've been told that I need to be a bit more rigorous - apparently I should do a chi-squared test to prove my hypothesis.

Could someone please give me a set of steps detailing how I would go about this?

Thanks.

-
I think you're supposed to do a $t$-test, not a $\chi^2$-test. See this. –  Ｊ. Ｍ. Sep 7 '11 at 18:24
@J.M. I believe that the t-test assumes a normal distribution. For my data nothing is assumed about the distribution. –  Griffin Sep 7 '11 at 18:38
Yes it does. Which means more work for you to do, i.e. a test for normality. Otherwise, I'm no longer up to date with what they use when the errors are non-normal. –  Ｊ. Ｍ. Sep 7 '11 at 18:40
Rigor definitely is needed, but a conventional chi-squared test probably won't achieve that, either. The sampling distributions of the slopes differ from one regression to another due to differences in the x coordinates, differences in the amount of data, and differences in the mean square error. –  whuber Sep 7 '11 at 20:02

## migrated from math.stackexchange.comSep 7 '11 at 19:22

This question came from our site for people studying math at any level and professionals in related fields.

This problem could be complex but it has a simple solution.

Let me start by explaining why the mean and SD of the estimated slopes are not relevant for testing whether all the true slopes are zero. Consider several illustrative hypothetical alternatives:

1. The true slopes are all zero, so that any nonzero estimates are the result of chance variation in the errors.

2. The true slopes all differ from zero, their estimation errors are small enough to show that every estimated slope significantly differs from zero, but the average slope is close to zero.

3. Some of the true slopes are zero but at least one is not, and for that one the estimated slope is significantly different from zero.

Note, in passing, that changing the units of measurement of the dependent variable will simultaneously change all estimated slopes and the SD of those slopes by the same amount, so that only the ratio mean(slope):SD(slope) is meaningful. In this case that ratio is approximately 1:10. Although this indicates there is a wide variation in estimated slopes, it cannot distinguish among any of these alternatives. That is why these two statistics are useless.

One problem is that these regressions are not necessarily related. They could involve different values of the independent variables. More importantly, the variation of the dependent variables could differ. Imagine (again hypothetically) a series of experiments in which a crude (but fast and cheap) measurement is made of the dependent variable, and then later is followed up by other experiments in which the measurement is made in a more precise way. We couldn't just dump the combined data into one regression model, due to the possibly large differences in distributions of the errors.

Another problem is that the residuals might not have Normal distributions. When each regression includes enough data (typically 30 points is "enough," but as always it depends), the sampling distribution of the estimated slope is still approximately normal, so the t-testing apparatus applies. With small amounts of data, or when some data have high leverage, the t-tests are suspect. However, that's a common problem having many cures, including using generalized linear models, re-expressing (transforming) the independent variables and/or the dependent variable, and other more specialized approaches. So let's assume that the regressions have been appropriately done. This means that we can trust the p-values for the slope tests.

Now we're off and running. The p-value for a single regression, under the null hypothesis that the slope is zero, will have a uniform distribution. Therefore the p-values obtained from each regression should behave like a set of independent draws from a uniform distribution. Small p-values suggest significant differences, so we are interested in whether there are more small p-values than would be expected by chance. (We should also be interested in whether the set of p-values really does look uniform: significant deviations from uniformity would suggest problems with the regressions or subtle violations of the null hypothesis.) A useful statistic, then, is the minimum p-value.

The axioms of probability immediately imply the minimum $p$ of $n$ independent draws from a uniform distribution has a CDF of $\Pr{[p \le t]} = 1-\Pr{[p \gt t]} = 1-(1-t)^n$. To test the null hypothesis that all slopes are zero at the $\alpha$ level (say $\alpha=0.05$, corresponding to $1-0.05 = 95\%$ confidence), we therefore follow this simple procedure:

(a) Let $p_0$ be the smallest p-value of all the $n$ regressions.

(b) Compare $1-(1-p_0)^n$ to $\alpha$. If it is smaller, conclude that at least one slope is nonzero. If it is not smaller, do not reject the hypothesis that all slopes are zero.

Note that this is algebraically the same as checking whether $p_0$ is smaller than $1-(1-\alpha)^{1/n}$. For small $\alpha$ this is close to $\alpha/n$. (The relative error made by this Bonferroni approximation is always less than $-\log(1-\alpha)/\alpha$, which is about $1+\alpha/2$. For $\alpha=0.05$ that's a relative error of about $1.025$, which for most purposes is negligible.) These considerations lead to a simple procedure indeed:

Reject the hypothesis that all slopes are zero only when the smallest p-value of the regressions is less than $\alpha/n$.

-

"To me, a layman, I would say that this shows that the gradients should be zero given the very small mean value and the tight standard distribution."

One can infer from the numbers you provided (using, e.g., Chebyshev's inequality) that the observed gradients, not only their mean and SD, were small. To the extent that the gradients derived from the data sets are independent observations (from similar distributions) of one underlying "true" gradient this is evidence that the parameter itself is small.

That the SD is ten times larger than the mean is consistent with the idea that the observed gradients are random errors around a true value of zero. It is also consistent with the mean being small (but nonzero) for systematic reasons, maybe similar to whatever reasons led to the suspicion that the mean is exactly zero, and observational noise larger than the true size of the mean.

To say more, additional information is needed about the number, size and nature of the data sets used to determine the gradients.

-
Much of this reply assumes that $10^{-4}$ is "small," but @John rightly points out this is a matter of units of measurement (and therefore is meaningless). The large SD (compared to the mean) shows that there is considerable variation of the individual slopes compared to their average. This might or might not be consistent with the true mean being zero. Finally, we cannot assume these slopes have a common sampling distribution; in fact, that's unlikely. Thus the SD actually tells us little about what's really going on. We need more information to say anything here. –  whuber Sep 7 '11 at 21:43
The OP called 1/10000 "small" and knows what units and meaning it has in his problem. Here the mean and SD have the same units and their ratio is dimensionless. It is not necessary to assume a single common sampling distribution, only similar variances (and if that is not the case, a small ratio of mean to SD still imposes strong constraints). More information is needed but as e.g. the Chebyshev calculation indicates one can say something, just a lot less than with full information. –  zyx Sep 7 '11 at 21:59
@whuber: re "that might or might not be consistent with the true mean being zero" -- how could it be inconsistent with the mean being zero? It is consistent with the mean being zero, and also consistent with the mean being nonzero but the observations of the mean having SD higher than the value of the mean. As was pointed out in the answer. Are you saying there are relevant classes of models where a small observed ratio (mean/SD) disfavors the hypothesis that the true mean is zero? –  zyx Sep 7 '11 at 22:10
I agree a process with zero slope could produce these statistics. But suppose, hypothetically, that one of two regressions has a slope of $0.0005 \pm 0.000001$ and another has a slope of $-0.00045 \pm 1$. Their mean is $0.00005$ and SD is $0.0005$, as in the question, but there is extremely strong evidence that one of the slopes is nonzero. That's why you cannot deduce anything about whether all slopes are zero from these statistics alone. Chebyshev doesn't tell you anything of relevance. –  whuber Sep 7 '11 at 22:11
As I have tried to explain, assumptions are needed to justify most of your conclusions, whether or not you were aware of them. So, yes, you "made no such assumptions," but you needed to. Also, perhaps because my example was communicated in a telegraphic manner, I think you may have misunderstood it. The value of 22000 (wherever it comes from) is not relevant. The $\pm$ values express standard errors of estimation, nothing more. If only a single slope has a low SD compared to its estimate, we should conclude that not all slopes are zero. –  whuber Sep 8 '11 at 13:29
show 3 more comments

The answer to your question would depend greatly upon a number of things you didn't mention, but what you're attempting is called a random coefficients analysis (RCA) and you'd want a t-test here against 0. It might be better to just generate a confidence interval around your coeffcient.

One might say that your standard deviation isn't small, it's rather large. It's 10x larger than the slope. Then again, I can't tell if any of the values are large or small because you haven't given a reasonable range of values that it could possibly be. Let's say the predictor in your regression is some very wide ranging value, perhaps mass of ships in grams. If your predicted value was something with a relatively small but meaningful range, like length of time for the hull to rust through in years, then your coefficient would be a very small number but still be a very large effect. So you need to give a lot more information before you can get a good answer.

-
"RCA" is so obscure I can't figure out exactly what it is from Web-accessible documents, but it seems to assume much more than given in the question. In particular, separate independent regressions need not even have comparable error variances (or even error distributions). So that's one more piece of information needed here: what, if anything, do all these regressions have in common? –  whuber Sep 7 '11 at 21:33
Yes, you are correct, I did sort of assume they had something in common, like came from similar power plants in different location, or something like that. RCA is sort of a replacement for repeated measures. It is possible that the regressions are unrelated. It was used extensively in early FMRI research because the amount of data made analysis using something like mixed effects intractable. It has been shown to be pretty good given a minimal amount of standard assumptions. –  John Sep 9 '11 at 1:06
I should say that there are still lots of ERP or FMRI researchers who use what could only be called RCA analysis. –  John Sep 9 '11 at 1:07
Oh, and I did a quick web search and found loads on random coefficients analysis. RCA has too many alternatives. –  John Sep 9 '11 at 1:13