How to assess whether students' responses to a test are affected by changing the format of the test

Question

We had students take a test with 20 questions and had them indicate their confidence levels for each question. In one test format, the confidence prompt was adjacent to their test question while in the alternate format, their responses to the test question and confidence had to be indicated on a separate form. The confidence responses are Likert scale data (not at all confident....highly confident, coded 1-5) while the test responses are dichotomous (correct/incorrect, coded 1/0). The tests only differ in the way the confidence survey was answered; test content and items are exactly the same.

We are unsure of whether the change in format might have had an impact on their confidence or test responses, but would like a simple method to check for this. There are definitely other factors that go into affecting their responses, but any recommendations on how we might be able to make any judgements about the effects of this format change?

Answer 1

The first major issue is whether the allocation of students to test-format conditions is independent of factors relevant to confidence ratings or test accuracy. Random allocation of participants to conditions would ensure this. In the absence of random-allocation, you would want to think about whether you have reasons to expect any systematic differences. In the absence of random-allocation, any observed group differences could readily be attributed to changes in the two groups of students. In particular, from a theoretical perspective, I think that merely asking a confidence question would not influence the accuracy of responses.

If you can assume independence as per above, then you could start by examining whether the means of the average confidence ratings or average accuracy difference between the two groups. Independent groups t-tests would be the standard choice.

More generally, there are a wide range of ways that the multivariate distribution composed of 20 confidence ratings and 20 accuracy ratings could vary. You would probably want to focus on the most relevant or theoretically like characteristics of interest. For example, a few statistics that come to mind include mean confidence, mean accuracy, standard deviation of confidence, and the within-person correlation between confidence and accuracy. For means differences you could use independent groups t-tests; for standard deviations, you could use Leven's test. For average within-person correlations you could also use independent groups t-tests.

Answer 2

This seems like the perfect application of differential item functioning. Differential item functioning is a test bias that leads to individual scoring differently on different tests, despite having similar levels of a latent trait/ability. For example, perhaps women who are equally as intelligent as men score poorly on an item on an IQ test because of cultural reasons. Alternatively, perhaps individuals who complete form B of a test (where items are in reverse order) score lower than those who complete form A of a test, despite being of similar capacity. This second scenario sounds very similar to what you are describing.

The best way to test for differential item functioning is by using item response theory. Try out jMetrik, it is free and fairly easy to use. It also includes the more common Mantel-Haenszel chi-square test for differential item functioning. In my experience results are always quite similar. You can use this to see whether the different forms of the test lead to different results.

If you want to combine results, you can use what is referred to as test/scale equating, which I think jMetrik can do as well. This allows you to compare scores on tests that should lead to same results and transform scores on one test to scores on another test.

Hope this helps.

Edit: Darn, I just noticed this question is a year old! Oh well, hopefully it helps someone.