4
$\begingroup$

Background

I'm currently trying to calculate a Cronbach's alpha statistic for an instrument that has participants take every item twice during the administration. We are using a speeded mental rotation paradigm where two stimuli are presented to the participant rotated at some angular disparity and the participant must make a judgement about whether they are seeing the same object that has just been rotated.

To get a reasonable estimate of average accuracy per item as a function of angular disparity, it is common (1) to have some a significant percentage (~50%) of your stimulus pairs be mismatched and (2) for the same stimulus to be repeated multiple times in a randomized presentation.

Problem

I need to calculate a reliability statistic for this instrument. Currently I'm treating each unique pair of stimuli as equivalent to an item on this test. I've shaped the data so that each unique test item has an "ItemIndex." This means that each presentation of the same stimuli pair has the same ItemIndex.

I'm familiar with using SPSS to restructure the data and run this analysis when each item only has one response per item in an administration, but I'm not finding a clear answer on how to handle the case where the same item is answered multiple times.

Sample Data

Table of sample data

The column labels stand for:

  • Participant: Unique ID given to each participant
  • ItemIndex: Identifier attached to each unique combination of stimuli (each unique presentation is considered at 'item' in our test).
  • Stimulus1, Stimulus2: Filenames of the stimuli that the participant saw during that trial. Note that matching stimuli pairs have the same ItemIndex
  • Accuracy: 1 if the participant correctly identified the stimuli as a match or mismatch, 0 if the participant didn't correctly identify whether the stimuli as match or mismatch
  • Presentation: 1 if this is the first appearance of this item, 2 if this is the second appearance of this item

Possible Solutions

  1. Can I somehow collapse identical items into a aggregated score on the item? This feels like it might violate the fact that the "Accuracy" column is really a categorical variable.
  2. Could I treat Presentation 1 of a stimulus pair as a distinct item from Presentation 2 of that stimulus pair? This feels like it wouldn't really be valid, though, given that these two items are really identical.

Any direction/resources to read would be appreciated!

Improve this question
$\endgroup$
4
  • $\begingroup$ I think that you should calculate alpha analogously with the way that the scale is scored. How is it scored? $\endgroup$
    – Jeremy Miles
    Jun 13, 2017 at 22:43
  • $\begingroup$ @JeremyMiles The items are scored as either correct ("1", e.g. stimuli Match and participant responds Match) or incorrect ("0", e.g. stimuli Match, but participant responds Mismatch). In my sample data set this is stored in the Accuracy column. $\endgroup$
    – dfd0226
    Jun 13, 2017 at 22:46
  • $\begingroup$ And then these are summed? $\endgroup$
    – Jeremy Miles
    Jun 14, 2017 at 16:23
  • $\begingroup$ Currently they aren't being added, but this is something I could easily do. I see you posted this below as well. I will give this a try now. $\endgroup$
    – dfd0226
    Jun 14, 2017 at 20:03

2 Answers 2

Reset to default
1
$\begingroup$

The data as described violations the assumptions that allow you to interpret alpha. You could calculate some sort of composite reliability, but I would collapse the data to the item level, so for each item a person scores 0, 1 or 2, and then calculate alpha.

Improve this answer
$\endgroup$
0
$\begingroup$

You do not want to find alpha. This is a case of test-retest reliability for a vey short interval (what Cattell called dependability).

Dustin Wood has a nice article discussing the issue of reliability of a test when you give the same item twice. (Dustin Wood and P. D. Harms and Graham H. Lowman and Justin A. DeSimone Response Speed and Response Consistency as Mutually Validating Indicators of Data Quality in Online Samples. Social Psychological and Personality Science, 2017.)

The next release of psych (1.8.2, coming soon) has a function to take repeated items like that into account. In the interim, you could try the multilevel.reliability function in psych but testRetest is (will be) better.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.