Statistically significant?

Question

I am new to statistics and so any help would be greatly appreciated!

I have done a study looking at the number of samples being sent to the lab for patients with UTIs in a GP practice. I then implemented a training program to teach the correct guidelines for sending a sample and although the sample size is very small I would like to know who to work out if the results are significant - (SD and p value etc)

Initially there were 21 patients with a UTI in a 3 month period. Of these 8 had a urine sample sent of which only 3 were done correctly.

After the training, there were 33 with UTIs in another 3 month period. of these 11 had a sample sent of which 9 were done correctly.

Looking at the percent this is a good improvement (37.5% to 81.8% of samples sent correctly) but I am aware the sample size is very small. Any help in finding if this is significant would be greatly appreciated!

Thanks

Answer 1

A run of your values for test of proportions (with and without Yates correction) is as follows:

>  prop.test(n=c(8,11), x=c(3,9), correct=F)

        2-sample test for equality of proportions without continuity correction

data:  c(3, 9) out of c(8, 11)
X-squared = 3.9095, df = 1, p-value = 0.04801
alternative hypothesis: two.sided
95 percent confidence interval:
 -0.84875961 -0.03760402
sample estimates:
   prop 1    prop 2 
0.3750000 0.8181818 

Warning message:
In prop.test(n = c(8, 11), x = c(3, 9), correct = F) :
  Chi-squared approximation may be incorrect


>  prop.test(n=c(8,11), x=c(3,9), correct=T)

        2-sample test for equality of proportions with continuity correction

data:  c(3, 9) out of c(8, 11)
X-squared = 2.2368, df = 1, p-value = 0.1348
alternative hypothesis: two.sided
95 percent confidence interval:
 -0.95671416  0.07035052
sample estimates:
   prop 1    prop 2 
0.3750000 0.8181818 

Warning message:
In prop.test(n = c(8, 11), x = c(3, 9), correct = T) :
  Chi-squared approximation may be incorrect

As you can see, the numbers are just significant without correction and not significant with correction.

If you want to assess whether the proportion of samples sent for cases of UTI changed with training, you can run following. These numbers also did not differ significantly:

>  prop.test(n=c(21,33), x=c(8,11),  correct=F)

        2-sample test for equality of proportions without continuity correction

data:  c(8, 11) out of c(21, 33)
X-squared = 0.1276, df = 1, p-value = 0.7209
alternative hypothesis: two.sided
95 percent confidence interval:
 -0.2150739  0.3103120
sample estimates:
   prop 1    prop 2 
0.3809524 0.3333333 

>  prop.test(n=c(21,33), x=c(8,11),  correct=T)

        2-sample test for equality of proportions with continuity correction

data:  c(8, 11) out of c(21, 33)
X-squared = 0.0042183, df = 1, p-value = 0.9482
alternative hypothesis: two.sided
95 percent confidence interval:
 -0.2540349  0.3492730
sample estimates:
   prop 1    prop 2 
0.3809524 0.3333333 

It is clear you need more numbers for such a study. A power analysis for sample size may be helpful: http://www.statmethods.net/stats/power.html

Answer 2

You could look at a two-sample proportions test (particularly if you seek a one-tailed test), or at a 2x2 chi square.

Here it is framed as a contingency table suitable for a chi-square:

          correct incorrect  Total
  Before     3       5         8
  After      9       2        11 

  Total     12       7        19

The small sample sizes should not be a big issue as far as appropriateness of the usual statistics; all the expected counts are reasonable. However, it will be an issue for power -- it looks like you may have a large effect, but you probably won't be able to tell it from just random noise.

The exact p-value you get will depend which test you want to apply, on any continuity-type corrections applied because of discreteness of the test statistic, and so on.