How do you determine an effect size where the variables have a maximum?

Question

I am reading a paper that looked at the change in test scores as a result of an intervention (pre-test versus post-test scores). The tests determine the skill level of students, and can range from 0 to 100% for any individual student. They used a t-test to look at the change resulting from the intervention, and used Glass's delta to determine the effect size. However, since the test scores had a maximum value (100%) and a large proportion of the pre-test obtained that maximum score (about 40% of the students), I suspect it violates the assumptions behind a t-test (result distributions don't seem to be normal). In particular, the students that got 100% in the pre-test can only go backwards in the post-test, irrespective of how much they learned and improved their skills during the intervention.

What is the appropriate statistical test to use for such experimental designs?

Also, the value of both Cohen's d and Glass's delta are their intuitive understanding to non-statistically educated people - you can say to your manager "this intervention improved the skills by an average of 20%". Other effect size methods (e.g. r^2) result in blank stares, followed by long and sometimes futile explanations. Does the method that you suggest have an intuitive understanding of how large the effect size is? If not, would it be appropriate to cut the students that scored above a set score (e.g. 80%) from the analysis and subsequently state "students with poor skills (below 80% in the pre-test) improved by an average of 20%"?

Answer 1

You are right that the assumption of normality is not met, and therefore one should interpret a paired t-test with care.

Given that your data is continuous, but censored at 0 and 1 I would expect some kind of ordered logit or probit model (as described here) would solve your problem at least partly, but the problem is that the data is paired. Some suggestions for that (but not definitive answers) are given here.

If you can actually use such a model you can calculate marginal effects and use those as a measure of effect size. Marginal effects have a relatively straightforward interpretation.

Answer 2

You are right - a t-test in this situation certainly violates the normality assumption. The true effect will be underestimated, and the power to detect it should be reduced compared to correct models. Truncated regression solves the problem by assuming truncated Gaussian error distribution. I'm not sure if it can explicitly incorporate paired data - still, in my opinion, with so many truncations accounting for those could be more important than accounting for the paired study design.