6
$\begingroup$

I have sampled 8 bags of a certain brand of candy to compare the color distributions of the candies. I have 4 bags for each size of bag, 8 oz and 1.9 lb. The bags were paired randomly. Here are my hypotheses:

$\ \ \ \ H_0: The \ distribution \ of \ each \ color \ of \ candies \ is \ equal \ in \ all \ sizes \ of \ bags.\\ \ \ \ \ H_A: The \ distribution \ of \ each \ color \ of \ candies \ is \ not \ equal \ in \ all \ sizes \ of \ bags.$

I then ran 4 chi-square tests for each pair of bags, generating 4 p-values. With an alpha level of .05, 3 of the pairs suggest I fail to reject my null hypothesis but one suggests I reject it. What is the best way to draw a conclusion from this? Should I overall fail to reject the null hypothesis because the majority shows this?

Improve this question
$\endgroup$
6
  • $\begingroup$ You state you have four bags of each size. On what basis, then, have you paired them? $\endgroup$
    – whuber ♦
    Commented Jun 5, 2017 at 17:40
  • $\begingroup$ There are 8 total bags, 4 8 oz ones and 4 1.9 lb ones. None of them are equal in size. $\endgroup$
    – Shivashriganesh Mahato
    Commented Jun 7, 2017 at 0:55
  • $\begingroup$ How, then, do you manage to pair them? There are 9 ways to construct a set of pairings of the 8-oz bags and the 1.9-lb bags: how did you select one of them? $\endgroup$
    – whuber ♦
    Commented Jun 7, 2017 at 13:15
  • 1
    $\begingroup$ It was randomized. $\endgroup$
    – Shivashriganesh Mahato
    Commented Jun 9, 2017 at 1:39
  • 1
    $\begingroup$ That is crucial information for answering your question. Please include it in your post. But why randomly pair the bags? That adds no information and precludes the use of more powerful techniques. What exactly is your null hypothesis? $\endgroup$
    – whuber ♦
    Commented Jun 9, 2017 at 14:46

3 Answers 3

Reset to default
11
$\begingroup$

If all of your null hypothesis are, in reality, true, then your probability of rejecting in at least one of your experiments is

$$ 1 - 0.95^4 \approx 0.19 $$

So there about a 20% chance you would find at least one rejection in your experiment, even if all of the bags had an equal distribution of colors. Not too unlikely; how you decide to act now depends on the costs of being wrong.

I suggest you eat 20% of the candy.

Wouldn't it be (1-.95)^4?

I think I got it right:

  • Probability of one experiment falsely rejecting: $0.05$
  • Probability of one experiment not falsely rejecting: $0.95$
  • Probability of all experiments not falsely rejecting: $0.95^4$
  • Probability of at least one experiment falsely rejecting: $1 - 0.95^4$
Improve this answer
$\endgroup$
3
  • $\begingroup$ Wouldn't it be (1-.95)^4? $\endgroup$
    – Shivashriganesh Mahato
    Commented Jun 4, 2017 at 23:08
  • $\begingroup$ @ZodiacZubeda Edited my answer. $\endgroup$
    – Matthew Drury
    Commented Jun 4, 2017 at 23:11
  • 4
    $\begingroup$ @ZodiacZubeda, for completeness: (1-95)^4 would be the probability that all bags falsely reject the null hypothesis (which is quite small). $\endgroup$
    – Mayou36
    Commented Jun 5, 2017 at 9:16
7
$\begingroup$

If you are trying to test if distribution depends on the bag -or, equivalently, if all bags are random samples from the same population- performing tests on pairs of bags is not going to work, because it can yield contradictory results -as you found- and because probability of type I errors is going to build up due to the multiple comparisons problem -as Mathew Durry's answer explains and as the XKCD comic demonstrates in a different context.

You can avoid this problem by performing a single test using all bags: a chi-square test for homogeneity, which will tell you whether there are significant differences between bags.

Please notice that most online examples of this test use just a pair of samples, but it works equally fine for more samples. Furthermore, the test is the same as the chi-square test for independence (just interpretation is a bit different), so you can find information under both names.

If homogeneity test shows that there are significant differences between bags, you might be interested on knowing between which bags there are significant differences. Then, paired tests can be useful, but to prevent the multiple comparisons problem to happen again, you need to make corrections. I would suggest Bonferroni correction because of its simplicity.

Anyway, if your bags are just random bags taken from a shop shelf, knowing which one is significantly different is uninteresting and the homogeneity test should be enough for your purposes.

Improve this answer
$\endgroup$
2
$\begingroup$

After you explain the results in the Results chapter, you can state in the discussion that one result was found significant. You can provide your interpretation of the results based on literature and suggest number of plausbile explanations to the reader.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.