0
$\begingroup$

I have a psychological test that consist of a set of binary items. It has a noticeable ceiling effect in the sum scores (essentially, a majority of participants get most items correct).

I also see some literature using that same measure, standardising and aggregating the sum scores with those of other measures, making a composite score for use in prediction. I have a hunch that this somewhat questionable, but can't for the life of me point at something concrete. What i'm wondering is:

  1. How would a "bad" indicator affect the overall measurement attributes of a composite score?
  2. Is it the ceiling effect problem "solveable" by standardisation?
  3. How is the issue affected by renorming?
  4. How correct is it to rescore by dividing the scale using quantiles? (I'm guessing there will be very few items differentiating betwen the high-scoring individuals, thus making the categirisation in the upper levels fairly arbitrary wrt. "true" latent ability.)

Or if there are more potential issues with it that is beyond my imagination.

Improve this question
$\endgroup$
8
  • $\begingroup$ Some of this will depend on what exactly you plan to do with the scores. You'll have ceiling effects if they are used as response in a linear regression, but you may use a model for ordinal regression or other nonlinear modelling. $\endgroup$
    – Christian Hennig
    Commented Jul 10, 2023 at 11:56
  • $\begingroup$ Tanks for the response! My approach would be to use it in a structural equation model, either as a predictor or as an outcome. Would that deal with it though? $\endgroup$
    – FredrikH-R
    Commented Jul 11, 2023 at 11:13
  • $\begingroup$ I don't think so. I don't know the details but standard structural equation models assume linearity as far as I know. Your data may be such that it doesn't do much harm, but I can't know this. $\endgroup$
    – Christian Hennig
    Commented Jul 11, 2023 at 14:42
  • $\begingroup$ Q1: What is a "bad" indicator? What makes it bad? $\endgroup$
    – Christian Hennig
    Commented Jul 11, 2023 at 14:44
  • $\begingroup$ So a basic measurement model won't solve it then. I suppose using an IRT model could work, as they only assume monotonicity. That requires item response data though, and typically you'd only have access to sum scores (long live paper and pencil). $\endgroup$
    – FredrikH-R
    Commented Jul 12, 2023 at 9:03

1 Answer 1

Reset to default
1
$\begingroup$

To what extent ceiling effects are a problem depends on what exactly you want to do with the score. Ceiling effects certainly cause issues if the score is used as a response in a model that assumes linearity (which implies unlimited growth potential of a variable). Normality will also be violated, maybe critically (even though this is assumed for residuals rather than the raw distribution of a variable). With use as predictor this is not necessarily a problem (linear regression doesn't involve model assumptions for predictors).

  1. I think a composite score only makes sense if the things to be put together make sense to be put together regarding meaning and interpretation. Otherwise it is hard to give any results regarding the composite score a meaningful interpretation.

  2. Standardisation will not help. Standardisation is a linear transformation, and fits of linear models are equivalent using linearly transformed data.

  3. Not sure what exactly you mean by "normalisation" but chances are what I wrote in 2 applies also here.

  4. Again not quite sure what exactly you mean here but just dividing by anything won't help (nor harm actually), see 2. You may mean something else though, namely using a new score in which certain values are merged?

That said, depending on knowing all the data, their meaning, and what exactly you want to find out, ceiling effects could ultimately be "mostly harmless", but based on the given information, nobody can say this.

Improve this answer
$\endgroup$
7
  • $\begingroup$ Thanks again for answering my questions, despite of my poor formulations (i feel i lack some of the stat. vocabulary, at least with fundamentals). The variables are absolutely assumed to go to infinity (at least for most practical purposes), and for the rest of the indicators in the composite variable, most measure well around the mean of the distribution. The main construct is theoretically solid. I’m a bit puzzled by your comment on linear regression, as i thought a basic linear model accomodated error in the outcome variable, not the predictors. But perhaps i just have things backwards. $\endgroup$
    – FredrikH-R
    Commented Jul 12, 2023 at 9:44
  • $\begingroup$ 3. Some norms used in the standardisation process was conducted on samples which the distributional mean were around the middle of the scale, which is not the case for samples from later studies (the centering i think not being around the raw scores of the sample mean, but the mean of the norming sample). I wonder if this compounds on the other issues. 4. Yes, i mean merging scores (going from a continuous variable to for example a dichotomous (low/high), or a standard nine scale. Sorry for being imprecise. $\endgroup$
    – FredrikH-R
    Commented Jul 12, 2023 at 9:44
  • $\begingroup$ @FredrikH-R Regression models error in the response as you correctly say, but its model assumptions also refer to the response (or rather the residuals which are derived from the response), so issues with the response distribution may be critical issues with model assumptions. $\endgroup$
    – Christian Hennig
    Commented Jul 12, 2023 at 15:22
  • $\begingroup$ @FredrikH-R Regarding 3, I can't comment on this without seeing a precise formal definition. Regarding 4, it depends on the actual data whether this helps. I'm not extremely optimistic but I can imagine situations in which it could. Of course you may want other kinds of models, for example logistic regression is you create binary data, or ordinal regression. $\endgroup$
    – Christian Hennig
    Commented Jul 12, 2023 at 15:25
  • $\begingroup$ Well that makes sense. It would be interesting to check the impact of the breach of model assumptions, and how creative data managment affects results. Do you have any suggestion for some light reading on the topic? (Although, general research on things like this probably have greek letters make up literally 30% of the total character count, which is ... well, nightmarish.) Anyway, thanks a lot for answering my questions here! It has been immensely useful for getting my head around this. $\endgroup$
    – FredrikH-R
    Commented Jul 17, 2023 at 10:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.