Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 6 down vote favorite
1

We generated repeated measures data from a sample of people evaluated at 4 timepoints in 2 groups. We wish to compare the groups over time. There are significant missing values. The questionnaire is very insensitive and has a score range from 1-30, however the vast majority of people are scoring 29 or 30. The numbers fall off exponentially below 29. Admittedly there is more of a downward spread in the measurements made at the first timepoint. Log-transforming (nor any transformation) has not made a difference. Notwithstanding the obvious rubbishness of the actual questionnaire, have people encountered this situation before? At the moment we are simply using 29 as a cutoff for a categorical analysis. Is there anything more elegant which can be done?

share|improve this question
4  
A question or scale that shows great ceiling effect is psychometrically bad one, obviously, because its discriminating power is so poor on the upper edge. You can't expect to make fine analysis out of rough instrument. Your decision to cut into 2 or 3 categories in such a situation is right, for me. – ttnphns Nov 9 '11 at 19:00
2  
I think your assessment is right, @ttnphns, but I wonder about the advice to use coarse categories. If there is any effect to be found here, it will appear precisely in those few responses that fall below 29. Lumping them together is likely to destroy any chance of detecting significant differences. Instead, we should direct our attention to procedures that are as powerful as possible, and that seems to mean using the actual data without unnecessarily categorizing them. – whuber♦ Nov 9 '11 at 21:16
3  
Sounds like a Tobit model see Wikipedia may be appropriate here. Essentially, the model assumes that there is a latent variable whose value is directly observed below a particular threshold (30 in your case) and censored if the latent value falls above that threshold. As suggested by @whuber, this makes use of all of the data without the need for artificial categorizations. – Wolfgang Nov 9 '11 at 21:49
@whuber, I'm fully aware and share your concern that categorizing will kill information taken by the left wing of the scale. But is that information valuable? Item or construct with great ceiling effect is not simply of poorly calibrated difficulty, - it is likely to measure not exactly what it was supposed to measure. Yes, the left wing is sensitive to some thing and measures it, but is this thing relevant? – ttnphns Nov 10 '11 at 6:08
@ttnphns Good questions, all of them, well worth considering in any reply here. – whuber♦ Nov 10 '11 at 7:10
show 1 more comment

1 Answer

up vote 2 down vote accepted

Is it the case that each individual's score is composed of the sum of 30 binary questions? If so, then you should analyze the raw data (1 or 0 for each question for each individual) genearlized additive mixed effects models, treating individuals as random effects, and specify a binomial link. For example (in R):

library(lme4)
fit1 = lmer(
    data = my_raw_data
    , formula = accuracy ~ (1|individual)
    , family = binomial
)

This would fit a model where there is only an intercept. If you have a between-individuals manipulation coded in a variable called "A", you could evaluate the amount of evidence for an effect of A by:

fit2 = lmer(
    data = my_raw_data
    , formula = accuracy ~ (1|individual) + A
    , family = binomial
)
(AIC(fit1)-AIC(fit2))*log2(exp(1)) #bits of evidence for an effect of A

Where "bits of evidence" refers to a likelihood ratio represented on the log-base-2 scale. Negative bits would represent evidence against an effect of A.

The ezMixed() function from the ez package automates the computation of such evidence metrics, and the ezPredict() and ezPlot2() functions facilitate obtaining and visualizing effects.

If I am wrong and the score does not represent the sum of 30 binary questions but instead the sum of some smaller number of likert-coded questions, you could recode the likert responses to binomial as I suggest here, then proceed as above.

share|improve this answer
+1 Perhaps you might also have a suggestion for when this kind of binary data is transformed into population based z-scores but the skew remains? i.e., a score of 30, for a person between 50-60 gives a z-score of 2.5. But this might require a separate question. – Matt Albrecht Nov 10 '11 at 3:05
1  
These are all really useful comments, thanks. The data are actually not 30 x 0/1 but Likert scales. It is a scale assessing activities of daily living. The problem is that it is designed for dementing patients and we're using it on depressed patients. The ceiling effect is caused by its insensitivity to the more subtle instrumental difficulties (financial management, shopping, complex tasks) that recovered depressed people have. – rosser Nov 10 '11 at 9:38
However, at baseline there's quite a spread below 30, so it is detecting "disability", however, when people return to "normal" functioning after treatment, they all arrive at 28-30 or thereabouts. – rosser Nov 10 '11 at 9:38
I wonder is there a way to combine these ... there doesn't seem to be an established way to do a repeated measures Tobit model in the package VGAM. That would seem to be the most elegant solution. Previously, the (awful) literature has used raw change scores across 2 timepoints, with presumably significant insensitivity due to regression to the mean. – rosser Nov 10 '11 at 10:17
censReg package with plm.data looks promising – rosser Nov 10 '11 at 10:46

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.