I have a specific example for an experimental design using a control/test group that I want to make inferences on. I want to test the completion rate for the same training program done online vs. offiline. I'd like a critique of both my experimental design and the inference procedures I plan on using:

I have two offices (out of many) where I am experimenting with the training delivery. Only those two offices will be the focus of the study, and I will combine the results from each office into the same sample for inference. So, I want to infer to my larger population of offices from these two offices. Due to financial constraints, the two offices are not selected randomly, but are regional and pretty close to one another.

I will be setting up a parallel design at the same two offices, with the in-person training being given to all new hires that are hired within one 90 day period (control) and the online training being given in another 90 day period to all new hires (test) within that period. So, for example for March 1 - May 30 all new hires receive the training in person, and for Jun 1 - Aug 27th all new hires receive the training online - no one receives both training methods and there is no overlap .

I have 95 new hires in the control group, and 100 new hires in the test group. 43 people complete the training in the control group, and 35 people complete the training in the test group.

So, my control proportion is about 45%, and my test proportion is 35%.

So, I want to test the hypothesis:

$H_0$ : $P{(control)} \leq P{(test)} $ vs.

$H_a$ : $P{(control)} > P{(test)} $

with $\alpha = 0.95 $, or at the 95% confidence level.

I carried this test out in R and received A p-value of 0.09, so I cannot reject the null at the 95% level and conclude that the control is less than or equal to the test. I have evidence to reject the null at the 90% level, however.

enter image description here

How trustworthy are my conclusions here given the experimental design and the data?

I know that proportions that are close to each other in the actual population take alot of data to estimate differences. I calculated that for a 10% difference in the population, $\alpha = 0.95%$ that I'd need a sample size around 385-400 for each group, so around 800 observations. Since I dont have that, should I use something like fisher's exact test?

If I use fisher's exact, the result is definitely non-significant:

enter image description here

Also, are there any distortions/errors that could be due to the experimental design? The new hires are about the same for both periods: 95 vs. 100. Is it safe to use the same offices at different times for each training method like I did? Furthermore, is it safe to generalize the results to the larger group of offifces given the design?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.