I have 7 survey questions that I asked students about regarding their online learning experience. Each question is on a 5 point Likert scale including -1 for not applicable. Now that I have the results, how do I create a scaled score? Do I change the not applicable into missing data and then input replacements? I ran a Cronbach's alpha earlier on each variable as well as a principal component analysis. The problem is, outside of determining which components should have specific items based on item weights, I am not sure how to best create an overall advising scaled score to compare with continued enrollment (binary variable).

Also, should it be expected that any scaled score I create would have a low phi correlation with continued enrollment since each of the individual items do not have significant correlations with the continued enrollment variable? Thanks for your advice!


Let's see if I can address your different questions.

NA in a score: You have a fundamental problem in your scale - one of the items does not apply to some of the observations. That implies two data generating processes or mechanisms you need to consider: 1) What makes the question applicable; 2) What makes a person select a given response option. I can think of a nunmber of ways to treat this, one complex and the other easy.

  • Use equating - create a scale for individuals that answered this item which includes the item, and then create a scale for individuals that did not answer the item, which will exclude the item. Then equate the two scales using the items in common. That is fairly drawn out and I would avoid it if possible.

  • Create scale without the item. Create the scale with and without the item, and correlate the two for individuals that have the item, and see how they align. If they are highly aligned, and the reliability is the same or very close the two, then you do the scale without that item. You lose information there of course.

  • Impute the item. If you believe the "NAs" are not really not applicable, but respondents selected them for laziness or other reasons, you could try to impute them.

  • Use FMLE. Full information maximum likelihood in a CFA I believe can deal with this missingness. I'm not sure how this can be done in IRT, but I believe it is possible. This I think assumes missing at random, so like the imputation solution, if the mechanism is truly different, then this option may not work.

  • Create a separate item about applicability. Make a new binary item, 1=item applies, 0= does not apply. Use this in the factor model. This would make sense if the applicability of the item is correlated with the underlying construct. This option is probably the least informative, but you could (and perhaps should) test it.

Low correlation with outcome measure: You have a hypothesis there and want to test it. Go for it, test it. You may find that, indeed, low correlation for the individual observables equals low correlation for the construct. That is informative - in statistics we often think that no relationship is useless and not interesting. Far from it - if you find that a validated and reliable measure of a construct is not related to an outcome of interest, that is highly informative - it could debunk an entire theory. Don't give up because you are worried about low correlation. Test it and see, and then update your theory. That's the only way to get smarter - only looking for things you already are pretty sure exist is not the best research agenda.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.