Background
I'm currently trying to calculate a Cronbach's alpha statistic for an instrument that has participants take every item twice during the administration. We are using a speeded mental rotation paradigm where two stimuli are presented to the participant rotated at some angular disparity and the participant must make a judgement about whether they are seeing the same object that has just been rotated.
To get a reasonable estimate of average accuracy per item as a function of angular disparity, it is common (1) to have some a significant percentage (~50%) of your stimulus pairs be mismatched and (2) for the same stimulus to be repeated multiple times in a randomized presentation.
Problem
I need to calculate a reliability statistic for this instrument. Currently I'm treating each unique pair of stimuli as equivalent to an item on this test. I've shaped the data so that each unique test item has an "ItemIndex." This means that each presentation of the same stimuli pair has the same ItemIndex.
I'm familiar with using SPSS to restructure the data and run this analysis when each item only has one response per item in an administration, but I'm not finding a clear answer on how to handle the case where the same item is answered multiple times.
Sample Data
The column labels stand for:
- Participant: Unique ID given to each participant
- ItemIndex: Identifier attached to each unique combination of stimuli (each unique presentation is considered at 'item' in our test).
- Stimulus1, Stimulus2: Filenames of the stimuli that the participant saw during that trial. Note that matching stimuli pairs have the same ItemIndex
- Accuracy: 1 if the participant correctly identified the stimuli as a match or mismatch, 0 if the participant didn't correctly identify whether the stimuli as match or mismatch
- Presentation: 1 if this is the first appearance of this item, 2 if this is the second appearance of this item
Possible Solutions
- Can I somehow collapse identical items into a aggregated score on the item? This feels like it might violate the fact that the "Accuracy" column is really a categorical variable.
- Could I treat Presentation 1 of a stimulus pair as a distinct item from Presentation 2 of that stimulus pair? This feels like it wouldn't really be valid, though, given that these two items are really identical.
Any direction/resources to read would be appreciated!