# Am I using the wrong statistical test?

I am using $\chi^2$ to analyze a set of data gathered over 4 time periods. The two variables are independent--each 'event' is categorized as 'good' or 'bad'. We want to prove that the intervention caused an increase in the good events and decrease in the 'bad' events, or at least that the increase in good events outpaces the bad events.

Here is a sample of what the data might look like:

       good    bad
m1     1035    2278
m2     1152    2643
m3     1189    2917
m4     1125    2974


A $\chi^2$ test returns 15 which would have me believe the result is significant, but the change doesn't look very significant at all. In fact, I don't think there is a case to be made that there wasn't any improvement at all. I think that this may be due to the fact that my $N$ varies in each measurement (though I thought $\chi^2$ controlled for that).

It also seems strange to me that if I simply move the decimal over two places (thereby changing my N but not the proportion of good results to bad) then I get $\chi^2=.15$ which is closer to what I would expect.

I know I need to change my approach drastically. How can I make this comparison? Should I be using two independent student's t-tests? Or maybe just use the correlation coefficient for each column and the coefficient of determination?

Thanks in advance for your help and please let me know if anything is unclear.

EDIT:

Here is a thought: should I set up benchmarks for expected improvement, something like 3-5% increase (proportional to $N$) in each measure? This way my expected values in my $\chi^2$ might more accurately reflect what I am trying to show.

EDIT for clarification:

the % of good events compared to bad over the measures:
31%, 30%, 28%, 27%

Clearly there has not been improvement with regard to the proportion of good to bad so what test should I be using to demonstrate this?

-
Actually, if the samples are independent, that's a highly significant difference. Consider m1 and m4: $p_1=0.312$ and $p_4=0.274$, but with those sample sizes, the std. error of $p_1$ is only about 0.008 and of $p_4$ about 0.007. Consequently the two are roughly 3.6 std. errors (of the difference) apart. –  jbowman Jun 27 '12 at 13:53
How exactly is the "intervention" reflected in the data? How can one identify "increases" in good and bad events from a table like the one presented here? –  whuber Jun 27 '12 at 14:09
Thanks for the replies. The workers that produce the data are rewarded based on the number of 'good' events they produce. They were also given a tool that helps them produce more good events. I still can't see why, when I have about the same mean during the entirety of the study, my test claims the improvement is significant. –  user12234 Jun 27 '12 at 16:49
add comment

## 1 Answer

I agree with @jbowman, taking your data at face value, you have a highly significant effect. This might be a good time to review what statistical significance is. The p-value of a statistical test gives the probability of finding data as extreme or more extreme than your data if the null hypothesis is true. Generally, people will believe that the null hypothesis is not true if the p-value is less than some threshold, $\alpha$. In practice, $\alpha$ is almost always arbitrarily set at .05. It is exceedingly unfortunate that this idea had been given the label 'significant'. That word automatically connotes importance in peoples' minds, however, importance is basically unrelated to statistical significance. The p-value simultaneously measures two things: the effect size, and $N$. The reason your situation seems perplexing is that you have a small effect, but an $N$ that is so large you get a highly 'significant' p-value anyway. Years ago, Kirk coined the term 'practical significance' to denote the idea of whether the effect was large enough to be meaningful. A Google search led me to this, which looks good upon skimming it and may be helpful for familiarizing yourself with the ideas.

On a different note, it is not clear what can or should be concluded from your data. It appears that you do not have a control group, or an estimate of the trend prior to the intervention. From a logical point of view, these facts make drawing any conclusions about the effect of the intervention problematic. Furthermore, I'm guessing that your observations are not independent, as they appear to be sequential measurements taken over time. Note that independence is one of the most important assumptions of the $\chi^2$ test. Thus, from a statistical point of view, using the $\chi^2$ test, or other simple statistical test, for inference may be problematic as well. There are ways of dealing with such situations, but they are more advanced. You may want to work with a statistical consultant. (Although if the data are not logically capable of answering your questions, it may not be worth it--unfortunately this case reminds me of a famous quote by the statistician Sir Ronald Fisher.)

-
I do have a baseline measure (which I should have identified). We then took subsequent measures as the intervention was implemented. Thanks for the clarification on independent observations- I understand now why the test was failing to demonstrate what I was seeing. Could I simply use the correlation coefficient and student's t? I just need to identify a trend and be able to defend intelligently the strength of that trend. –  user12234 Jun 27 '12 at 21:17
Also, in response to your quote from Sir Ronald Fisher, to quote Monty Python, "I'm not dead yet!" –  user12234 Jun 27 '12 at 21:24
The existence of the baseline is comforting, but without a control group, I think you would need to be able to estimate the prior trend. Think of it this way, what if the proportion of good results had been going down beforehand at a faster rate, & after the intervention it is still going down, but at a slower rate? Then, the intervention was positive, even though the post intervention slope is negative. I don't see how you can rule that out. Moreover, standard correlations & t-test will be subject to the same problems, so I don't think that will help you. I hope it works out. –  gung Jun 27 '12 at 21:51
You're absolutely right. Thankfully, the study is only for performance improvement and doesn't have to be perfectly scientific. We began the study because we knew this behavior was a problem so we'll assume it was already dismal and stable (which I understand, is a big assumption). Thanks for your help. –  user12234 Jun 27 '12 at 21:58
@user12234 gung is right that you really needed prior info or a control group to make statements about the impact of the intervention. But - you do have plenty of information to estimate the trend over the course of the study. I think a logistic regression (proportion good events ~ time, weighted by total events) would let you say pretty definitively that the proportion of good events is decreasing in a way that is not due to chance (but again, nothing about the intervention). –  Matt Parker Jun 27 '12 at 22:14
add comment