1
$\begingroup$

I am trying to test if there is a significant effect of test method versus test score. I have two groups of people, the first group contains $10$ people who did a test using an online version versus a second group, which contains $15$ people who did a test using a sit-down version.

For each group, they did the exact same test twice on two separate occasions. Hence, for example, for the group of $10$, I have their test scores for the first time they did a test, and the second time they took the exact same test. The problem with the tests is that the online version has a different overall score than the sit-down version.

I would like to take the change in scores for the online version from the first time they took the test to the second time, and compare it to the sit-down version.

Since the overall scores are different, I am trying to look at the percentage increase in the scores. However, the problem here is that a test percentage score increase from $50\%$ to $70\%$ is greater than a test percentage score increase from $80\%$ to $100\%$, so there is a scale problem.

Is there a hypothesis testing method I can use to compare score increase from the first test to the second, across different test types here?

$\endgroup$
  • $\begingroup$ What is your goal in using Hypothesis Testing? What questions would you like answered in your data (last sentence kind of hits this, but a bit more specific could help)? Is your data coded in a way that tells you if they took Online or Sit-down test first? $\endgroup$ – Kunio May 8 '18 at 0:55
  • $\begingroup$ I am trying to see two things: 1) Use a hypothesis test that gauges if there is a significant difference in scores between the sit-down and online test. 2) Trying to gauge if there is a difference in the first to second test for the sit-down test only. I.e., Does taking the same test twice result in significantly better scores? The caveat here is that people who start at a lower score should experience greater relative change differences, so does the risk ratio work better here? What hypothesis test works here? $\endgroup$ – user321627 May 8 '18 at 0:57
  • $\begingroup$ To be clear, are you stating that the two groups started at unequal levels on the first administration of the test? Hence their relative growth will be different due to differences in possible growth? $\endgroup$ – Bryan May 8 '18 at 1:32
  • $\begingroup$ Yes that is exactly it! $\endgroup$ – user321627 May 8 '18 at 2:41
  • $\begingroup$ Sounds pretty hopeless. The experimental set-up is likely too much of a mess to dip much with the data. $\endgroup$ – Björn May 8 '18 at 4:52
2
$\begingroup$

There are no easy solutions when comparison groups that start an experimental study at different levels. Likely these groups were not randomly formed, so differences other than those measured may also exist further confounding interpretation of results.

One option is to use ANCOVA (or regression) and treat the pre-test (first administration) as a covariate to provide adjusted – predicted – second administration scores that show what the scores, and group differences, ‘might’ be if both groups started at the same level for the first administration. Unknown confounding variables represent a serious problem with this approach if groups were not randomly formed; they likely differ in other important ways so any adjustment on the second administration scores are highly suspect and interpretation should be treated with caution.

Some use gain scores, but as you realize, again, if groups start at different levels on the pre-test, then gain scores can be affected by floor or ceiling effects, and differential gain is not so easy to interpret. Repeated measures analysis is also possible, but it too suffers from the same shortcomings noted above when unequal groups are studied.

Normalized gain, symbolized as g, is one attempt to address the problem with gain score analysis with unequal group starting points on pretests. Normalized gain appears to be used in physics education research. Below is a description on normalized gain that I wrote elsewhere.

Normalized gain represents the proportion improvement of what could be improved from pre-assessment to post-assessment. For example, suppose a student scores 40 of 100 correct on a pre-assessment. The amount of improvement possible from the pre-assessment score is 100 - 40 = 60. Suppose further that this student scores 70 on the post-assessment. The increase from pre-assessment to post-assessment is 30 points, and this 30 points represents a 50% increase over the pre-assessment score in terms of what could be gained, i.e., gain of 30 divided by the possible gain of 60 is 30/60 = .50 or 50%. Similarly, a student who obtains a pre-assessment score of 90 has only 10 points of possible improvement on the post-assessment to obtain a maximum score of 100. If this student scores 93 on the post-assessment, that represents a 30% increase over what could have been gained, i.e., a 3 point increase from post-assessment to pre-assessment (93 - 90 = 3) which is divided by the maximum possible gain of 10 points (maximum score minus pre-assessment score is 100 - 90 = 3). Thus, this second student has a normalized gain score of 3 / 10 = .30 or 30%.

The formula for normalized gain is provided by Bao (2006):

$$ g = \frac{(PostAssessment\ Score) - (PreAssessment\ Score)}{(Maximum\ Score) - (PreAssessment\ Score)} $$

If using percentage scores, the formula presented by Colettaa and Phillips (2005) applies:

$$ g = \frac{(PostAssessment\space \% - PreAssessment\space \%)}{(100 \% - PreAssessment\space \%)} $$

where post-assessment % is percentage correct on the post-assessment and pre-assessment % is the percentage correct on the pre-assessment.

By using the normalized gain score the focus shifts from absolute test score gains to relative gains.

One problem with normalized gain occurs when a post-assessment score is lower than the corresponding pre-assessment score (i.e., post-assessment < pre-assessment). In this situation the interpretation of normalized gain fails and the calculated values no longer represent the portion gain or change relative to what could be gained. When post-assessment < pre-assessment, Marx and Cummings (2007) proposed the following formula:

$$ g = \frac{(PostAssessment\space \% - PreAssessment\space \%)}{ PreAssessment\space \%} $$

This formula provides a change score that presents the proportion loss from the pre-assessment starting position. This interpretation is more consistent with normalized gain, although the focus with this formula is loss from where one started.

Another problem occurs when pre-assessment and post-assessment scores are both 100%. In this situation, one has achieved the utmost score for both tests. Therefore, the normalized gain would be 100 for this student.

I don’t know whether normalized gain is widely accepted, but it is one approach that could be considered.

References

Bao, L. (2006). Theoretical comparisons of average normalized gain calculations. Physics Education Research, 74, 917-922.

Colettaa, V. P., & Phillips, J. A. (2005). Interpreting FCI scores: Normalized gain, preinstruction scores, and scientific reasoning ability. American Journal of Physics, 74, 1172-1182.

Marx, J., & Cummings, K. (2007). Normalized change. American Journal of Physics, 75(87-91).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.