I have a psychological test that consist of a set of binary items. It has a noticeable ceiling effect in the sum scores (essentially, a majority of participants get most items correct).
I also see some literature using that same measure, standardising and aggregating the sum scores with those of other measures, making a composite score for use in prediction. I have a hunch that this somewhat questionable, but can't for the life of me point at something concrete. What i'm wondering is:
- How would a "bad" indicator affect the overall measurement attributes of a composite score?
- Is it the ceiling effect problem "solveable" by standardisation?
- How is the issue affected by renorming?
- How correct is it to rescore by dividing the scale using quantiles? (I'm guessing there will be very few items differentiating betwen the high-scoring individuals, thus making the categirisation in the upper levels fairly arbitrary wrt. "true" latent ability.)
Or if there are more potential issues with it that is beyond my imagination.