3
$\begingroup$

Consider the following scenario. I ran an experiment with 200 trials, each with a different stimulus. Each subject did exactly 200 trials. The subject responds with a single number anywhere between 5 and 50. The correct answer also ranges between 5 - 50. For each subject that did the experiment, I compute a single value, $V$, for that person. This value $V$ uses computations that use the expected answer and the observed answer for each trial. That is all the experiment does. It allows me to find $V$ for a subject.

I was asked to find Cronbach's alpha for this particular experiment. How do I do it? I see the formula on Wikipedia, however the denominator is $K-1$ and in my case, $K=1$, so how would I find Cronbach's Alpha? Should I find a single Cronbach's alpha value for the experiment as a whole, or is each subject getting an alpha value? I couldn't find many good online resources to learn about Cronbach's alpha value, so if anyone has any good suggestions, I would love to see links to places I can learn this stuff. Thanks!

I will be using R, and I have a CSV file where column 1 is the subject, column 2 is the expected answer, and column 3 is the subjects answer. There are 200 rows.

Improve this question
$\endgroup$
12
  • $\begingroup$ If you only have one measure per subject, you can't measure within-subject reliability. But if each subject does the 200 trials (is that what happened) you can use a measure of reliability if you have that individual trial data. What data do you have? What are the responses you are measuring - are they binary (e.g. success/failure) or more than binary (e.g. Likert items, multiple choice answers)? $\endgroup$
    – Michelle
    Jan 22, 2012 at 22:05
  • 1
    $\begingroup$ I'd echo @Michelle's first sentence. Cronbach's alpha indicates internal consistency; how can you find the consistency between a single response and itself? So I'd ask why you were looking into Cronbach's alpha and whether there is something else that better captures what you're looking for. $\endgroup$
    – rolando2
    Jan 22, 2012 at 22:21
  • $\begingroup$ And echoing @rolando2, what was the purpose of your experiment so what are you trying to find out? $\endgroup$
    – Michelle
    Jan 22, 2012 at 22:37
  • 1
    $\begingroup$ @CodeGuy Please update your question accordingly, and describe very precisely the experimental setup you used. This way, we could clean up that long series of comments that has become very hard to follow. $\endgroup$
    – chl
    Jan 23, 2012 at 18:08
  • 3
    $\begingroup$ We still don't know enough detail about the experimental set-up. What was the stimulus exactly (e.g. number of pixels presented in 300ms on a computer screen, pressure applied to a subject's hand in mmHg, windspeed of a puff of air blown onto the subject's eyeball)? What question was the subject asked in order to produce the response, and how was this measured (e.g. visual analog scale of pain intensity, a number they clicked on a computer screen). I believe that is what @chl meant by "describe very precisely" in his comment above. $\endgroup$
    – Michelle
    Jan 23, 2012 at 20:35

2 Answers 2

Reset to default
6
$\begingroup$

The formula for Cronbach's alpha is: $$ \alpha =\frac{K}{K-1} ( 1-\frac{\sum_{i=1}^{K}\sigma^2_{Y_i}}{\sigma^2_X}) $$ Here, K is the number of different items you administered to each subject. Sometimes items are different questionnaire items designed to measure the same underlying construct. In your case, it sounds like each item is a separate run of the experiment.

In order to calculate Cronbach's alpha, you need to put your data in "wide format" (as Michelle mentioned). This means that each of the 200 measurements needs to have its own column/variable. So your columns would be SubjectID, Answer1, Answer2, ... , Answer 200.

I'm not sure where your expected answer column comes in. Cronbach's alpha is used to test the consistency of answers to each other, not to some true value, because the true value being measured is latent (i.e., unknown). I suppose you could calculate the correlation between the mean of the answers and the expected answer to see how well people did. Or calculate the times they answered exactly the way you expected and call that their score.

As has been suggested in the comments, it doesn't seem to make sense to calculate alpha on the raw scores. If they are only meaningful in relation to the expected answer (i.e., as indicative of how well a subject does on this particular test), you need to use adjusted scores that actually quantify how well a subject did. If an answer of 50 with an expected answer of 25 doesn't mean the same thing as an answer of 50 with an expected answer of 45, then calculating alpha on the raw scores is meaningless.

To calculate Cronbach's alpha using R, read the CSV file into a dataframe, reformat into wide format, then run cronbach.alpha on only the answer columns (assuming your columns for subject and values are called SubjectID and Score):

x.long <- read.csv(file="myfile.csv")
library(reshape2)
x.wide <- dcast(x.long, SubjectID ~ Score)
library(ltm)
cronbach.alpha(x.wide[,-1]) # remove SubjectID in first column
Improve this answer
$\endgroup$
7
  • $\begingroup$ Thank you so much. This is so helpful. So just to understand the formula, here, K is 200? $\endgroup$
    – CodeGuy
    Jan 23, 2012 at 0:28
  • $\begingroup$ Yes, K=200. But if you use cronbach.alpha() that will compute K for you. Would be useful to calculate it by hand and using that function to make sure they are the same. Note I just adjusted the code to remove the first ID column from the data frame which should not be submitted to the cronbach.alpha call. $\endgroup$
    – Anne Z.
    Jan 23, 2012 at 0:31
  • 1
    $\begingroup$ Please note that the interpretation of Cronbach's alpha assumes that the scale is unidimensional, and I have hard time imagining that this holds for 200 items! $\endgroup$
    – chl
    Jan 23, 2012 at 14:39
  • $\begingroup$ These are 200 runs of an "experiment" (asking the subject for a number between 5 and 50??) not 200 different questionnaire items. They seem to be intended to be roughly identical measures of some single underlying construct--what that construct is, I don't know--but assuming unidimensionality may be reasonable. $\endgroup$
    – Anne Z.
    Jan 23, 2012 at 15:08
  • 3
    $\begingroup$ I cannot see how a response that can vary between 5 and 50, with an expected value on each trial that is being ignored, will produce a high reliability for a test. Say I presented subjects with a random number of white pixels on a black screen, between 5 and 50, for 300ms each time and you had to answer how many white pixels there were, for 200 trials, a Cronbach's alpha on those raw responses is not the correct analysis to use. $\endgroup$
    – Michelle
    Jan 23, 2012 at 17:38
3
$\begingroup$

Update with proviso: it is impossible to answer the question without knowing the detail of what occurred in the experiment, as the risk of providing incorrect advice is very high. Depending on what actually happened in the experiment, it is possible that both of my suggestions below are incorrect. This update is from my perspective that "software is stupid" - so long as the input is numbers (in most cases) you'll get an answer on any analysis you do, the problem being that the analysis may not have been appropriate, and therefore the analysis "answer" is wrong. This assumes that the desired ultimate outcome is to provide an answer that is appropriate to the method and data, and therefore may recommend a solution that is different to the one asked in the original question.

Information on Cronbach's alpha is here and any standard statistics package will calculate this for you. You will need to use the individual trial data to do the calculation so K=200. Assuming you're going to be using R or similar, you'll need the data in wide format. If you believe that you may have training or acclimatisation effects in your data, you may wish to consider a split half reliability alternative, for which you will still need all the individual data points. The point of doing this is to tell you whether V is an appropriate summary statistic, given your data.

Is there a reason why you think that something like a mean and standard deviation won't be informative? Internal reliability is normally performed when you don't administer an identical stimulus repeatedly to a person (e.g. when they answer a questionnaire, because a questionnaire doesn't consist of the same question repeated a number of times). If you're presenting them with the same stimulus repeatedly, I'm not sure why you would wish to use a reliability measure. I'm now not convinced that a mean and SD is the right approach either, however depending on what the experiment actually did with the subjects an analysis based on the difference (maybe relative rather than absolute given the response range) between the expected answer and the subject's answer could be a way forward.

Improve this answer
$\endgroup$
1
  • 2
    $\begingroup$ One other form of reliability (i.e., apart from that indirectly assessed by internal consistency) is temporal stability (typically assessed with intraclass correlation). $\endgroup$
    – chl
    Jan 23, 2012 at 14:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.