1
$\begingroup$

Assume the following research experience. A group of researchers went to a school. On the first day, they gave a questionnaire about a topic for the students to fill. The next day, the researchers gave the same students a lecture about that topic. The study finished on the 3rd day when the researchers gave the same questionnaire and collected the answers from the same students.

The questionnaire collects age groups, gender, and other descriptive categorical variables. The scale of the questions about the topic was the Likert-type. Also, data are not paired.

The group of researchers now wants to compare, e.g., age groups, to conclude how effective the intervention was. For example, for a given question, did age group 15–18 reveal differences before and after intervention? To accomplish this, we are doing Mann-Whitney U-tests on every subsample of interest.

Is this the correct approach? Should the p-value be corrected (e.g., Bonferroni correction)?

$\endgroup$
2
  • 1
    $\begingroup$ Keep in mind that a pre-post design, without an element of randomization, is one of the weakest of all designs. And it depends on having no missing data. $\endgroup$ Commented Aug 10, 2022 at 11:16
  • 1
    $\begingroup$ You might want to construct a single model that includes gender, age, and other demographic information. If appropriate, you might use multiple linear regression for this (general linear model). If you want a nonparametric approach, it sounds like aligned ranks transformation anova would work for your situation. In either case, you can then make the post-hoc comparisons that are of interest to you. $\endgroup$ Commented Aug 10, 2022 at 12:35

1 Answer 1

3
$\begingroup$

You are asking about the most fundamental question in causal inference. How do I know whether an observed difference between people who got the treatment and people who did not (or people before and after they got a treatment) was actually CAUSED by the treatment.

This is such a big question that you really should read a book or article on causal inference to get a sense of what the issues are...it's too big a subject to discuss here. But briefly...

-The kind of research design you are proposing here is called a pre-post design with no control group. Any intro to causal inference will discuss the strengths (which are few) and weaknesses (which are many) of this approach.

-In short, the biggest challenge for this sort of design is the possibility of a "history" effect. That is, what if the students just got better over the course of the few days for reasons that had nothing to do with the lecture? Maybe they just needed some time to absorb the material and the lecture itself was worthless. Generally, the way we address those sorts of concerns is by giving pre and post intervention questionnaires to a set of students who did NOT get the lecture. This is what's called a difference in differences design. But there are other ways to try and deal with this threat as well.

-Spending time worrying about correcting p values is a distraction. A p value just tells you how confident you can be that a result from a random sample can be generalized to a larger population. They do nothing to tell you whether your causal inference is correct. If you have a badly designed study that doesn't address threats to validity the result might be "significant" at p<.000001 (even after a correction) but the result will still be bogus. You need to do something to control for threats to causal inference. That might be a research design decision (like getting a control group) or through some sort of analysis procedure (like regression analysis). Once you sort out all that then you can think about whether a given test is appropriate or not. But you have a lot of more important stuff to deal with first.

-The only question about "statistical significance" you should probably care about at this stage is how roughly many students you will be surveying. If it's only a few dozen then you probably won't have enough statistical power to detect a significant difference even if there is a big difference between pre and post. On the other hand, if you have a few thousand, then almost any difference you see will be "statistically significant," even if it's so small that it's substantively irrelevant, and arguments about the appropriateness of different statistical tests will be irrelevant.

-Forget about looking for sub group difference until you've established that the lecture had a "main" effect at all. If you've established that the lectures had an effect on students in general, you can then do various things to try to see if the effect was larger or smaller for different groups. My personal approach to this would be to include an interaction term in a regression model, but there are other ways to do this as well.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.