Test for significance when looking at 2 proportions?

I am very new to statistics, and I would appreciate any help.

I have two independent groups of physicians, A and B.

Each physician group is measured on how successful they are at certain quality-of-care measures for patients, all of which are nominal data (screened or not screen, at goal or not at goal).

I am trying to figure out if the rate of screening in each group of physicians is statistically significant. From what reading I've done, possibly a chi squared test (with group A the expected outcome and group B the observed outcome)? Or a TOST (two one-sided tests)? Can a T-test be done if my data are not averages?

• A t test does not seem to be a good choice here. // It's unclear what 1-sided test you'd do. Maybe it's agreed that the 'standard' screening rate should be above half. Then for Gp A you might test $H_0: p=.5$ vs, $H_a: p > .5.$ Under $H_0,$ the number $X$ screened has $X \sim\mathsf{Binom}(1000, .5),$ and one can show (by exact computation in R or by normal approx.) that $P(X \ge 600) \approx 0,$ so it is clear you'd reject $H_0$ at the 1% level. The hard part lies in knowing what 'standard' rate to use for such a test. // My Answer shows how to compare Gps A and B with each other. Jan 6, 2021 at 1:49

You can run a chi-squared test (or Fisher test, if the N is small) on the 2x2 confusion matrix of doctor group x outcome measure. This will tell you if there is a statistically significant association between the grouping of doctors and the outcome of the quality-of-care measures. For this analysis, you don't need the proportions directly, but rather the joint count of events (# screened by A, # not screened by A, # screened by B, # not screened by B).

Just a note - you mention in the question that you want to "figure out if the rate of screening in each group of physicians is statistically significant". The analysis described here will tell you if the rates of screening in each group are significantly different from one another (which is the typical analysis), but you could also do a different test to see if the screening rates are statistically significant by themselves, which would typically imply a test to see if the rates are significantly different from zero (although that's probably not what you're going for).

Thank you so much Nuclear Hoagie! That makes sense. So my contingency table would look like the example below?

The analysis you described ("if the rates of screening in each group are significantly different from one another") is exactly what I'm looking for, not the other situation you described :)

Edit: I am the original poster, username is somehow different!

• Please register &/or merge your accounts (you can find information on how to do this in the My Account section of our help center), then you will be able to edit & comment on your own question. Jan 5, 2021 at 20:55

A test in R that can compare the screening rates of the two physician groups is prop.test, as below. I have not used the continuity correction because of the relatively large sample sizes. There is a significant difference between observed screening rates $$0.6$$ and $$0.7;$$ the P-value is nearly $$0.$$ If screening rates were truly equal, it would be extremely rare to observe such different rates between such large samples.

prop.test(c(600, 1400), c(1000,2000), cor=F)

2-sample test for equality of proportions
without continuity correction

data:  c(600, 1400) out of c(1000, 2000)
X-squared = 30, df = 1, p-value = 4.32e-08
alternative hypothesis: two.sided
95 percent confidence interval:
-0.13640472 -0.06359528
sample estimates:
prop 1 prop 2
0.6    0.7


Note: The test performed here is essentially the same as described by NIST. The NIST version gives a normal test statistic which is the square root of the chi-squared statistic in the output from R.)

A very slightly different version of this test is implemented in Minitab, which also shows results from Fisher's Exact Test [mentioned in the Answer of @NuclearHogie(+1).]

    Test and CI for Two Proportions

Sample     X     N  Sample p
1        600  1000  0.600000
2       1400  2000  0.700000

Difference = p (1) - p (2)
Estimate for difference:  -0.1
95% CI for difference:  (-0.136405, -0.0635953)
Test for difference = 0 (vs ≠ 0):  Z = -5.48  P-Value = 0.000

Fisher’s exact test: P-Value = 0.000


Note: Fisher's exact test is often used when counts are too small for normal or chi-squared approximations to be reliable, but using R we can compute probabilities for larger samples. The test statistic for a one-sided test is hypergeometric, based on observed totals in your $$2\times 2$$ table. With P-value very near $$0$$ the Fisher Exact Test also finds strong evidence against equal screening rates.

If there are 2000 screened subjects out of 3000, and I observe 1000 from Group A, what is the probability I see as few as 600 screened cases among them. That is the P-value of the one sided test. (The P-value for a 2-sided test is approximately double the P-value for a one-sided test.)

Roughly speaking, if the two groups screen subjects at the same rate, I would expect to see about $$(2/3)(1000) \approx 667$$ screened cases among my $$1000.$$ In fact, I see only $$600.$$ What are the chances of seeing $$600$$ or fewer?

phyper(600, 2000, 1000, 1000)
[1] 3.302131e-08


Here is a plot of the relevant probabilities from the hypergeometric distribution.