0
$\begingroup$

Say I’m interested in running an experiment. I want to see if student using collaborative techniques (say we pick one random technique) affects test scores. So, obviously the null hypothesis is that there is no change in test scores and the alternative hypothesis is that there is some significant change from employing this treatment. I’ll give my experimental design idea and I want to know what would be the test to use if this experiment were to actually take place.

So say there are two courses and the students have roughly the same level in math (or whatever as long as it’s the same subject matter). Class A is the control group and class B is the treatment group. In the treatment group, the students are split into many smaller groups since normal classes have 20 + students. There are, say, 4 exams (and they’re the same for both groups) given during the year.

How do I test if the treatment made a difference? I’m thinking a One-Way ANOVA. Or a two sample t-test? Is the fact that there are smaller groups in Class B a problem or could just average all those exams and compare it to Class A’s? What about the fact that there are 4 exams throughout the year? I feel like that’s problematic. Would it be best to pick one exam, say the final exam to test whether one or not the treatment made a difference? Are there other things I should be concerned about? Is One-Way ANOVA the right way to go here?

Improve this question
$\endgroup$
3
  • 2
    $\begingroup$ Hi, can you clarify what does "split into many smaller groups" mean? Will these smaller groups have somehow different teaching/learning/treatment experience with regards to each other and with regards to the control group? Why is the control group not divided into these smaller groups? $\endgroup$
    – Sointu
    Nov 28, 2022 at 12:43
  • 1
    $\begingroup$ You need to clarify your post. Sample size? 4 exams during the year indicates this is repeated-measures. Maybe add that tag? $\endgroup$
    – kjetil b halvorsen ♦
    Nov 29, 2022 at 18:43
  • $\begingroup$ @Sointu it's because in the control group, there is no collaborative learning being administered. For sample size, we can assume there are 30 students in both the control and the experimental group. $\endgroup$
    – Jama
    Nov 30, 2022 at 1:11

1 Answer 1

Reset to default
1
$\begingroup$

(I'll put this as an answer because of comment space limits, but it's not a comprehensive answer)

So the smaller groups within the treatment group all get different treatment/teaching? In that case, perhaps you can treat each of these groups + control as levels of your "treatment" factor. E.g. if you have 4 of these smaller groups within Class B, then you'd have a 5-level treatment factor. Of course, this would make a very unbalanced design so you couldn't use ANOVA. You could probably use a multilevel regression with the 5-level factor as predictor and (if there is a time component and participants are tested several times across time) participant as random effect, and I guess you'd need to add a "class" random effect too.

I think that as you haven't run the study yet, you might want to consider the design more closely before collecting data. Perhaps you could try recruiting equal number of participants for each of the "subgroups" of your treatment, including the control group? Perhaps even so that you'd sample several classes (more than 2) and assign some number of students from each class in each of the subgroups and also in control group? This way, you could account for possible differences between classes and their possible effects on treatment -> outcome relation, and you'd get more data and maybe also a balanced design.

Improve this answer
$\endgroup$
4
  • $\begingroup$ The smaller groups within the treatment group get the same teaching. I'm just looking to see whether working in groups of say, 5 helps with students' grades. Would it be best to simply calculate and compare the final exam grades since those exams are comprehensive and would be an indicator of how students learned the material overall $\endgroup$
    – Jama
    Dec 2, 2022 at 4:15
  • 1
    $\begingroup$ I see. But if you want to study the effect of treatment on some learning outcome, you should also divide the control group to smaller teaching groups, otherwise how can you know whether differences between treatment and control group are due to the treatment or to the size of teaching group? As for your second question, that's probably something you're the best expert of, but I personally would use objective grades if they were available (of course you can also test the effects of treatment on other variables, such as self-reported learning experience or some such). $\endgroup$
    – Sointu
    Dec 2, 2022 at 9:54
  • $\begingroup$ I have multiple groups because it would be somewhat extreme to use the whole class a group. I just want to see if working in groups is useful. Not too worried about the size of the group right now. So in theory it would be best if I just have set groups for the entire semester right? So that's there's no variability in scores attributed to working with different people $\endgroup$
    – Jama
    Dec 2, 2022 at 21:48
  • $\begingroup$ It's difficult to comment on the design of the study from the outside, you're probably the best judge of that as you know how the teaching and all the details will go. I just think it's important to make sure other relevant characteristics (such as teaching group size) are not confounded with your experimental treatment. $\endgroup$
    – Sointu
    Dec 3, 2022 at 9:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.