When to normalize learning? I'm trying to determine the effect of three types of learning on a group of subjects. I have their pretest scores and posttest scores.
The current goal is to determine which intervention reduced the percentage of wrong questions before the learning correctly answered after: That is to say, (Post - Pre) / (100 - Pre).
However, doing this per subject means that people who did very well on the pretest have a huge impact on efficacy. Averaging the Post/Pre per intervention corrects for this.
But I have no clue of the tradeoffs involved.
 A: There is really no good solution to your problem.  You have to think about what kind of improvement would be equivalent "learning" for people with pretest score of, say, 5, 50, and 95. Your current method says that 5 to 47.5 is the same amount of learning as 50 to 75 and 95 to 97.5 - half of the remaining questions. The method you are suggesting states that 5 to 7.5 is the same as 50 to 75 or 95 to an impossible 145.5 - a 50% increase. Do these really imply the same amount learned?
The problem with the Post/Pre ratio (and the Post-Pre differences), that the range of possible outcomes depends on the Pre score. With the original method, anybody can get essentially any adjusted score from 0% to 100% (though for high Pre values discreteness will be substantial), but for the ratio only students with Pre score below 50 could double their result. So high Pre scores will be pulling down the averages. 
An additional complication is that the variability of the scores is different at the low and high end of the scale than in the middle. If somebody's true score should be 5 or 95, than on a repeat test (without any actual learning) you would expect their score to be within +/- 5 points, but for somebody with a score of 50, the variability is +/- 10 points.
Three possible alternatives are:


*

*Use the ratio of the odds of a correct response: Post/(100-Post) divided by Pre/(100-Pre) - at least every ratio is possible for everybody. This would imply, for example, that 5 to 9.5 is the same as 50 to 66.7 and 95 to 97.5 - doubling of the odds.

*Look at how many standard deviations they changed. For a score of n, the standard deviation is sqrt(n*(100-n)/100). This would imply that a change from 5 to 10 is the same as 50 to 60 and the same as 95 to 100.

*Use a regression model of the Post scores on learning type adjusting for Pre scores. This tries to estimate the effect of changing the learning type when Pre score is the same. This lets the data guide the amount of adjustment needed, but can be difficult to explain afterward.
