Hypothesis test two probabilities are different

Question

I am trying to understand hypothesis testing, I've been following this tutorial: http://www.r-tutor.com/elementary-statistics/hypothesis-testing/two-tailed-test-population-proportion, but the combination of double negatives is getting a little confusing.

In my data, I have two separate populations (from the same overall population), for simplicity I shall call them "class1" and "class2". I want to check the probability that the probability of something occuring within these two classes is NOT EQUAL.

If I understand this correctly, this means I have to present the null hypothesis that they ARE EQUAL, and test if I can reject it.

Using R and the prop.test method:

class1 has a total of 1551 observations class2 has a total of 1446 observations

the probability of something occurring in class1 is 52.7% (817 observations) the probability of something occurring in class2 is 58.2% (842 observations)

On the surface, these probabilities do look different, so - looking good.

prop.test(817, 1551, p=0.582) gives: p-value = 1.159e-05

prop.test(842, 1446, p=0.527) gives: p-value = 2.849e-05

So, from what I can gather, this means that my probabilities ARE different, so I reject the null hypothesis. Is this correct?

Some questions:

1. Does it matter what the distribution of my data is?
2. What is the significance level of the test (is it the p-value, if so which value do I use?)

Thanks

Answer 1

Correct method for this should be by using all numbers in one test as follows:

> prop.test(c(817, 842), c(1551, 1446))

        2-sample test for equality of proportions with continuity correction

data:  c(817, 842) out of c(1551, 1446)
X-squared = 9.1169, df = 1, p-value = 0.002533
alternative hypothesis: two.sided
95 percent confidence interval:
 -0.09175405 -0.01932406
sample estimates:
   prop 1    prop 2 
0.5267569 0.5822960 

Distribution is generally not an issue in test of proportions. The P value is 0.002533

If you want to test one set of values with a given proportion, you should use binomial test:

> binom.test(817, 1551, p=0.582)

        Exact binomial test

data:  817 and 1551
number of successes = 817, number of trials = 1551, p-value = 0.00001188
alternative hypothesis: true probability of success is not equal to 0.582
95 percent confidence interval:
 0.5015559 0.5518565
sample estimates:
probability of success 
             0.5267569 

> 
> binom.test(842, 1446, p=0.527)

        Exact binomial test

data:  842 and 1446
number of successes = 842, number of trials = 1446, p-value = 0.00002477
alternative hypothesis: true probability of success is not equal to 0.527
95 percent confidence interval:
 0.5563847 0.6078723
sample estimates:
probability of success 
              0.582296 

The P values are similar to what you got with prop.test. These also gives confidence intervals of both proportions. There are no means here, only proportions and they are significantly different.

Read more about these tests in R using commands ?prop.test and ?binom.test