I have data that looks like this:

control = [0,0,1,0,1,1,0,1...] treatment-1= [1,0,0,0,1,1,0,1...] treatment-2= [1,1,1,1,1,1,0,1...]

The experiment was whether or not a certain return policy (control = no fee, t1 = 5 dollar fee, t2 = 8 dollar fee) would effect whether or not a person bought an item. Each person was exposed to only one treatment group. There are 300,000 samples for each treatment group.

What is the best way to test for significance of treatment? My first inclination was to run ttest's between pairs of 2 treatments, but my p-values were 0, which made me suspicious. Further research indicated this may be a proportions problem and maybe a chi square test is preferred, but I do not really understand why.

Please advice on how to test if there are significant differences in outcomes based on treatment. Thanks.


As you expected, the chi-squared test seems fit for your problem. The null hypothesis would be $$ H_0: p_1=p_2=p_3 $$ where $p_i$ is the proportion of people in group $i$ who decided to buy an item. The alternative would be not $H_0$.

  • $\begingroup$ Thanks. I've been doing more reading on chi-squared and I am confused. From what I understand, chi-squared tests the likelihood of any difference between the actual data and the expected data resulting from chance. Is this correct? If so, how does this apply to my scenario? I have no expected data, just actual data from 3 treatment groups. $\endgroup$
    – connor449
    Feb 9 '20 at 23:30
  • $\begingroup$ @connor449 You right there. I should've been clearer. Your problem is to test homogeneity among different groups in which you can use the chi-squared test as well. Please see this posting: mse.redwoods.edu/darnold/math15/spring2013/R/Activities/…. $\endgroup$
    – inmybrain
    Feb 9 '20 at 23:37
  • $\begingroup$ It's referred to as expected, but it's not your prior expectations. It's what's expected based on the row and column totals. The expected value for a cell is the row total times the column total, all divided by the number of observations. You're then checking how much your results differ from what's "expected" given the overall distribution of the data. In your case, for example, the expected value for not buying in the control is the total number of people in the control group times the total number of people who didn't buy an item, divided the total number of people observed overall. $\endgroup$
    – Todd Burus
    Feb 9 '20 at 23:43
  • $\begingroup$ Thanks for the responses. I think I understand. I created 3 2x2 contingency tables in python, one with control x t1, one with control x t2, and one with t1 x t2. I then computed chi-squared test. Can you aid me in interpreting the results? Control x t1 results (chi square: .30, p:.58. Control x t2 (chi square:.02, p:.88), t1 x t2 (chi square: 1.40, p:.23). I expected the control x t2 difference to be significant, but the p is .88, the highest. My results would make sense if the chi and p values were flipped...Is there any kind of relationship between p and chi square value? $\endgroup$
    – connor449
    Feb 10 '20 at 0:05
  • $\begingroup$ Two things. First, consider making a 3x3 table and testing them all together (this is what was suggested in the original solution). Second, yes, there is a relation between your chi-square value and the p-value. In fact, the p-value is determined by your chi-square value and the degrees of freedom in your test. A p-value is not something that exists on its own but only in relation to the appropriate distribution and test. There is no way these could have been "flipped". $\endgroup$
    – Todd Burus
    Feb 10 '20 at 0:27

Since I think you're interested in which method has the biggest effect and not just "did any method differ from the expectation", I would do pairwise difference of proportion tests with a family-wise error rate (FWER) correction (e.g. Holm's).

Alternatively, you could use the Marascuillo Procedure

Even if your results are significant however, the size of the effect is also important.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.