2
$\begingroup$

For the experiment, 12 people were invited. All participants were pre-tested (10 tasks with 0/1 marks). Then we randomly divided all the participants into two equal groups of 6 people: experimental and control. A treatment was conducted and all participants were post-tested (again 10 task with 0/1 marks).

Results of tests:

data = {'e0':  [6, 5, 5, 5, 6, 0, 0, 0, 1, 0], # experimental group, before a treatment 
        'c0':  [6, 5, 5, 5, 2, 2, 1, 0, 2, 0], # control group, before a treatment 
        'e1':  [6, 6, 6, 6, 6, 5, 2, 6, 2, 2], # experimental group, after a treatment 
        'c1':  [6, 6, 0, 6, 0, 2, 3, 3, 4, 2]} # control group, after a treatment

It is required to test a treatment had a positive effect on the experimental group.

Question. What is a statistical test suitable here?

My attempt is:

$H_0: \mu_0 = \mu_1$ vs. $H_a: \mu_0 < \mu_1,$ where the $\mu_i$ are the respective means for pre- and post-tests in the experimental group.

But in hypothesis statement I don't use the control group results.

Improve this question
$\endgroup$
2
  • $\begingroup$ The 10 tasks are obviously not interchangeable: there seems to be 4 hard tasks and 5 easy tasks. Note that simple tasks are not very informative because of a ceiling effect. The 0 in for the third task in the c1 group looks suspicious. The fact that controls do poorly on the fifth task is also interesting but maybe is due to chance because there are only 6 subjects per group. $\endgroup$
    – dipetkov
    Commented Jun 6, 2022 at 19:59
  • $\begingroup$ Because of this difference among the tasks, it's not clear what the two parameters $\mu_0$ and $\mu_1$ represent. Since your data consists of successes and failures, it's more natural to think in terms of probability of success and to let each task has its own probability of success. Perhaps both the tasks and the subjects can be modeled as random effects. You'd need the full data for this analysis. $\endgroup$
    – dipetkov
    Commented Jun 6, 2022 at 19:59

1 Answer 1

Reset to default
1
$\begingroup$

Because it's a small sample, this question and answer(s) are relevant. You're correct that you don't want to conduct a paired samples t-test on only the experiment group, because a comparison to a control group is important to your inference.

You could conduct an independent samples t-test on the post-task scores which would mean that the $H_0$ is the control group and $H_1$ is the experiment group. However, that does not leverage the baseline scores so if the groups randomly differed at baseline it could lead to faulty conclusions.

One option would be to calculate a difference score for each participant and conduct an independent samples t-test on the difference scores between the two groups.

There are a variety of issues to consider, in the linked post as well as other similar resources that address the assumptions of the t-test so I will not elaborate here. But there are certain conditions that you would want to check in order to be confident in the results of the t-test.

Regardless, this is a very small sample for a between-groups experimental design so it would be good to interpret the results cautiously.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.