1
$\begingroup$

I tried using chi-squared test, but it is giving me odd results,

For example:

Let's say that I am testing whether behavior between male and female users is the same. Males only bought 3 units of the first item, while females bought 5 units of every item.

males <- matrix(0,100)
females <- matrix(5,100)

males[1,1] = 3

males_and_females = cbind(males, females)

print(prop.test(males_and_females))

Result:

data:  males_and_females
X-squared = 186.7387, df = 99, p-value = 2.376e-07

It is telling that we can reject the null hypothesis that two samples proportions are the same. To me it is very strange that the test is telling us something with high confidence when there are only 3 observations for male users. This certainly feels wrong to me. I am probably using the test for what it is not meant for. I went over the implementation of chi-squared test, and I see that the high chi-squared comes from dividing by tiny expected values associated with zeros in matrix a.

It looks like R is suspicious of what I am asking it to do:

Warning message:
In prop.test(males_and_females) :
  Chi-squared approximation may be incorrect

My question: is there a version of adjusted chi-squared test that addresses the above issues? Or is there another test that handles this better? Or any other suggestions on how to test whether the proportions of two samples are the same when one sample has lots of zeros.

Edit:

print(chisq.test(males_and_females, simulate.p.value=TRUE, B=1e5))

Rusult:

    Pearson's Chi-squared test with simulated p-value (based on 1e+05 replicates)

data:  males_and_females
X-squared = 186.7387, df = NA, p-value = 7e-05
$\endgroup$
  • $\begingroup$ Have you tried something like chisq.test(a_and_b, simulate.p.value=TRUE, B=1e5)? $\endgroup$ – whuber Nov 20 '14 at 19:17
  • $\begingroup$ Can you state explicit null and alternative hypotheses? $\endgroup$ – Glen_b -Reinstate Monica Nov 20 '14 at 21:33
  • $\begingroup$ @Glen_b, my null hypothesis is the same as it would be for chi-squared test, that proportions of males[i] = females[i] for all i`, and the alternative is that this is not true. $\endgroup$ – Akavall Nov 21 '14 at 15:32
2
$\begingroup$

The answer is correct, please read the Details section in ?prop.test. You have more than two groups (you have, in fact, 100 groups). In 99 of those groups, you had 0 successes and 5 failures (P=0). In 1 of those groups, you had 3 successes and 5 failures (P=3/8). The hypothesis being tested is that all of the groups share the same probability for success... and this hypothesis is (rightly) rejected.

Try typing just prop.test(a_and_b) (i.e. without print), and you'll see all the groups being treated as independent samples.

If you reduce the number of P=0 groups, you'll start to see the test-statistic drop (and similarly the p-value rise):


> prop.test(a_and_b[1:10,])

    10-sample test for equality of proportions without continuity
    correction

data:  a_and_b[1:10, ]
X-squared = 17.8875, df = 9, p-value = 0.0365
alternative hypothesis: two.sided
sample estimates:
 prop 1  prop 2  prop 3  prop 4  prop 5  prop 6  prop 7  prop 8  prop 9 prop 10 
  0.375   0.000   0.000   0.000   0.000   0.000   0.000   0.000   0.000   0.000 
$\endgroup$
  • 1
    $\begingroup$ I don't think it was possible to give an answer I was asking for, but your post helped my understanding, so I am accepting the answer. Thank You. $\endgroup$ – Akavall Nov 21 '14 at 18:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.