3
$\begingroup$

I am having trouble finding clear answers to these specific questions about calculating Cronbach's alpha. For my dissertation study, I conducted a teaching intervention. I administered a pretest and posttest to investigate increased knowledge after the intervention. I developed the test items and I also developed parallel items to minimize testing effect (so participants didn't answer the exact same questions at pre- and posttest). I also had a control group who did not experience the intervention.

I am using alpha to measure the internal consistency of six subscales on the test and the internal consistency of the entire test. Each subscale consists of 8 multiple choice items that are supposed to measure one subtopic that I taught in the treatment. These are all parts of a larger topic, which is why I also want to look at the alpha for the whole test.

Here are my questions:

  1. Do I calculate and report alpha for the treatment and control groups together, separately, or only treatment group? It seems like combining them would be OK for the pretest, but after the intervention, aren't they different populations since one group has experienced the intervention? I expected the treatment group to improve and the control group to stay about the same. (And that is what actually happened.)

  2. Do I calculate and report alpha at pretest and posttest separately, combined, only posttest, only pretest? It seems like the pretest would contain random guesses if the participants didn't know the material before the intervention. So I'm not sure it makes sense to test internal consistency on that data.

The test items are multiple choice questions of knowledge (not Likert items). I have coded the responses as correct or incorrect. I have already run the alpha analysis on all these different possibilities, but I don't know what I am supposed to report. My advisor also doesn't know.

If anyone has a reputable citation, that would be ideal, but I am also very grateful to know your own practices and reasons.

Thank you!

$\endgroup$

1 Answer 1

2
$\begingroup$
  1. Do I calculate and report alpha for the treatment and control groups together, separately, or only treatment group?

Together.

  • a It seems like combining them would be OK for the pretest, but after the intervention, aren't they different populations since one group has experienced the intervention? I expected the treatment group to improve and the control group to stay about the same. (And that is what actually happened.)

That's an empirical question. However, if the intervention changed the reliability of the underlying scale, that's a problem. Because you're no longer assessing the same thing in the treatment and control group with the scale.

Example: Your outcome is happiness. Smiling is an indicator of happiness. This is one of your indicators and it correlates with your other measures of happiness. Your intervention affects happiness, but does not affect smiling. Now the reliability will be lower in the experimental group, because one of the indicators is lower quality.

This is (IMHO) very unlikely to happen. And if you wanted to detect it happening, you'd need a vast sample size.

  1. Do I calculate and report alpha at pretest and posttest separately, combined, only posttest, only pretest? It seems like the pretest would contain random guesses if the participants didn't know the material before the intervention. So I'm not sure it makes sense to test internal consistency on that data.

If the participants didn't know the material and were guessing, why bother giving them the test? If that was the case, your value of alpha would be low and you should discard the test.

Perhaps your test didn't measure the construct you were interested in, and actually measured their ability to guess. (For example, it's often the case in multiple choice tests that the shortest answer is more likely to be the correct answer [or sometimes the longest].) If someone used that strategy, the test will be reliable, but will lack validity. Hopefully you would find that the correlation between pre- and post-tests was low, and this would tell you that you might as well discard the pre-tests.

Depending on your sample size, time available and inclination, there are a lot of more sophisticated analyses that you can do, for example I would be considering looking at factorial invariance across and within groups.

I don't have a reference. But I've calculated coefficient alpha a lot of times.

$\endgroup$
8
  • 1
    $\begingroup$ Thanks for your reply! Let me try to clarify. I am not measuring a construct, like happiness. I am giving a knowledge test with right and wrong answers. I would expect responses to change after instruction. The pretest establishes a baseline to quantify those learning gains. The control group accounts for the possibility of maturation or testing effects. If the treatment and control groups perform comparably on the pretest and only the treatment group's scores significantly increase, then that suggests that the teaching intervention was effective in helping students learn the content. $\endgroup$
    – user405908
    Jan 31 at 5:02
  • 1
    $\begingroup$ What you said about participants guessing resulting in low alpha is what I have been arguing to my advisor. When I compare alpha between these four testing conditions (pre-post; control and treatment), they are not the same. It's possible that this is because the test is not reliable. However, the mean scores are low in all conditions except treatment group posttest. There is a statistically significant increase for treatment scores. So I'm trying to figure out what I should report and satisfy my dissertation chair. I'm not even convinced alpha is appropriate, but I've lost that battle $\endgroup$
    – user405908
    Jan 31 at 5:18
  • $\begingroup$ Good questions. Alpha has its limits, and if you are willing to go deeper analytically, then item response theory (IRT) should be an enormous help. IRT allows you to assess, for each item, the role of guessing; the item's difficulty; and the item's ability to discriminate between lower and higher levels of understanding of the content. One needn't go "full-throttle" and master all the nuances to derive benefit from some use of IRT. $\endgroup$
    – rolando2
    Jan 31 at 11:02
  • $\begingroup$ Your question and comments have convinced me that alpha is worth reporting only at post-test: probably for treatment, control, and combined, and you will interpret each somewhat differently along the lines you've alluded to. $\endgroup$
    – rolando2
    Jan 31 at 11:09
  • 1
    $\begingroup$ @user405908 "he said he always reports pretest only" - yeah, that's depressing. We do it this way because that's the way we've always done it, without thinking about why we do it, seems like bad science to me. $\endgroup$ Jan 31 at 17:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.