I have an experiment that compares the impacts on students' learning (as measured by their performance on a pre and posttest) of two versions of a classroom intervention: D and R.
I did an ANCOVA with condition (D and R) as the IV, posttest scores as the DV, and pretest scores as the co-variate. There was no effect of condition on posttest scores, nor was there an interaction between condition and co-variate.
Here's where I'm most unsure of my approach. To look at these data in another way, I divided students into two groups based on their pretest scores, and called them high vs. low prior knowledge (PK) groups. I ran a t-test to compare these groups' mean posttest scores, and found that within the D condition, high PK students did significantly better on the posttest than low PK students. There were no differences between high and low PK students within the R condition. There were also no differences between high and low PK students between the D and R conditions, which I suppose confirms the non-significant interaction found in the ANCOVA.
A reviewer of my manuscript said that it's inappropriate to manipulate the groups in the way that I did because it amounts to looking for an interaction, where the ANCOVA should serve this purpose. However, I'm wondering whether there's still value in having created these high and low PK groups, because the t-test did reveal a meaningful group difference within a condition, if not between conditions.
In case it's relevant to know, I also used used these high and low PK groups in a t-test, because I was curious about students' gains from the pretest to the posttest. The t-test found significantly greater gains among the D students compared to the R students. However, the effect size was low to moderate.
Finally, I did a t-test to compare the mean pre-to-post gains of high and low PK students. This test found that within the D condition, high and low PK students made similar gains. Within the R condition, low PK students made significantly greater gains than high PK students.
So I'm wondering whether the way I manipulated the data to create these high vs. low PK groups has some value, and if so, how I should justify this approach. Alternatively, if the analyses related to these groups is all around inappropriate and should be removed, why is that? Without them, the findings become much less interesting...