Am I using the wrong statistical test?

Question

I am using $\chi^2$ to analyze a set of data gathered over 4 time periods. The two variables are independent--each 'event' is categorized as 'good' or 'bad'. We want to prove that the intervention caused an increase in the good events and decrease in the 'bad' events, or at least that the increase in good events outpaces the bad events.

Here is a sample of what the data might look like:

       good    bad 
m1     1035    2278
m2     1152    2643
m3     1189    2917
m4     1125    2974

A $\chi^2$ test returns 15 which would have me believe the result is significant, but the change doesn't look very significant at all. In fact, I don't think there is a case to be made that there wasn't any improvement at all. I think that this may be due to the fact that my $N$ varies in each measurement (though I thought $\chi^2$ controlled for that).

It also seems strange to me that if I simply move the decimal over two places (thereby changing my N but not the proportion of good results to bad) then I get $\chi^2=.15$ which is closer to what I would expect.

I know I need to change my approach drastically. How can I make this comparison? Should I be using two independent student's t-tests? Or maybe just use the correlation coefficient for each column and the coefficient of determination?

Thanks in advance for your help and please let me know if anything is unclear.

EDIT:

Here is a thought: should I set up benchmarks for expected improvement, something like 3-5% increase (proportional to $N$) in each measure? This way my expected values in my $\chi^2$ might more accurately reflect what I am trying to show.

EDIT for clarification:

the % of good events compared to bad over the measures:
31%, 30%, 28%, 27%

Clearly there has not been improvement with regard to the proportion of good to bad so what test should I be using to demonstrate this?

Answer 1

I agree with @jbowman, taking your data at face value, you have a highly significant effect. This might be a good time to review what statistical significance is. The p-value of a statistical test gives the probability of finding data as extreme or more extreme than your data if the null hypothesis is true. Generally, people will believe that the null hypothesis is not true if the p-value is less than some threshold, $\alpha$. In practice, $\alpha$ is almost always arbitrarily set at .05. It is exceedingly unfortunate that this idea had been given the label 'significant'. That word automatically connotes importance in peoples' minds, however, importance is basically unrelated to statistical significance. The p-value simultaneously measures two things: the effect size, and $N$. The reason your situation seems perplexing is that you have a small effect, but an $N$ that is so large you get a highly 'significant' p-value anyway. Years ago, Kirk coined the term 'practical significance' to denote the idea of whether the effect was large enough to be meaningful. A Google search led me to this, which looks good upon skimming it and may be helpful for familiarizing yourself with the ideas.

On a different note, it is not clear what can or should be concluded from your data. It appears that you do not have a control group, or an estimate of the trend prior to the intervention. From a logical point of view, these facts make drawing any conclusions about the effect of the intervention problematic. Furthermore, I'm guessing that your observations are not independent, as they appear to be sequential measurements taken over time. Note that independence is one of the most important assumptions of the $\chi^2$ test. Thus, from a statistical point of view, using the $\chi^2$ test, or other simple statistical test, for inference may be problematic as well. There are ways of dealing with such situations, but they are more advanced. You may want to work with a statistical consultant. (Although if the data are not logically capable of answering your questions, it may not be worth it--unfortunately this case reminds me of a famous quote by the statistician Sir Ronald Fisher.)