3
$\begingroup$

I have an experiment that compares the impacts on students' learning (as measured by their performance on a pre and posttest) of two versions of a classroom intervention: D and R.

I did an ANCOVA with condition (D and R) as the IV, posttest scores as the DV, and pretest scores as the co-variate. There was no effect of condition on posttest scores, nor was there an interaction between condition and co-variate.

Here's where I'm most unsure of my approach. To look at these data in another way, I divided students into two groups based on their pretest scores, and called them high vs. low prior knowledge (PK) groups. I ran a t-test to compare these groups' mean posttest scores, and found that within the D condition, high PK students did significantly better on the posttest than low PK students. There were no differences between high and low PK students within the R condition. There were also no differences between high and low PK students between the D and R conditions, which I suppose confirms the non-significant interaction found in the ANCOVA.

A reviewer of my manuscript said that it's inappropriate to manipulate the groups in the way that I did because it amounts to looking for an interaction, where the ANCOVA should serve this purpose. However, I'm wondering whether there's still value in having created these high and low PK groups, because the t-test did reveal a meaningful group difference within a condition, if not between conditions.

In case it's relevant to know, I also used used these high and low PK groups in a t-test, because I was curious about students' gains from the pretest to the posttest. The t-test found significantly greater gains among the D students compared to the R students. However, the effect size was low to moderate.

Finally, I did a t-test to compare the mean pre-to-post gains of high and low PK students. This test found that within the D condition, high and low PK students made similar gains. Within the R condition, low PK students made significantly greater gains than high PK students.

So I'm wondering whether the way I manipulated the data to create these high vs. low PK groups has some value, and if so, how I should justify this approach. Alternatively, if the analyses related to these groups is all around inappropriate and should be removed, why is that? Without them, the findings become much less interesting...

$\endgroup$
2
$\begingroup$

I'm inclined to agree with your reviewer that this is a bad idea, for two reasons.

Firstly, you haven't explained why it is reasonable to split your continuous variable (pretest score) into a binary one. It is almost always a bad idea to bin (i.e. split) continuous data, as it involves throwing away useful information. Binning into two blocks is an extreme case of it.

Secondly, even though you may have done this with the best of intentions, this is bad use of p-values and the null hypothesis statistical testing framework. What you have done goes by many names: HARKing (Hypothesizing After the Results are Known), p-hacking, or more broadly and less pejoratively, the garden of forking paths. Essentially, after finding non-significant results one way, you changed a few things, ran a new analysis, and then found a significant result in one subgroup. There are a huge number of ways that one could tweak an analysis, and if you only stop when you find a significant result, you're essentially guaranteeing that you will find something significant. This massively biases any p-values you obtain and renders them meaningless. This is likely a major contributing factor to the replication crisis in science. These two sites allow you to play with simulations and real data to get a feel for how a basic version of this process biases results and inferences (note that the general idea described above is a bit more broad than the issues you can explore in these).

However, it is possible to learn from data in a way that avoids these pitfalls. You can

  1. switch to a Bayesian framework and avoid p-values altogether (although a similar issue arguably arises with the choice of prior); combining this with multilevel modelling may prevent many such erroneous inferences.
  2. preregister your analyses before your project, so that any p-values you obtain are not guided by decisions you make after examining your data.
  3. distinguish between hypothesis testing and data exploration, with p-value calculations avoided entirely for the latter. Data exploration can (and ideally should) then be used to inspire follow-up hypothesis tests with new datasets; the p-values in the new analyses will be more reliable.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.