I'm looking at the appropriateness of using the mean to summarize some Likert-type items, as opposed to taking some factor analysis approach. Ideally I'd like to use the mean, since it would be the easiest solution.
There are several (1,2,3) threads on related topics, and the general message I got from them was that researchers generally use the mean, although the appropriateness of doing so is disputed.
My question is different, and relates to how I can tell whether it's appropriate to use the mean given particular aspects of the survey under consideration.
What are the most appropriate considerations that could drive such a decision? I've made a list of candidates below, but I am not sure whether they are sensible ones, or whether my suggested ways of assessing the considerations are appropriate.
Look at whether the items are correlated with each other. I could perhaps throw out items with near-zero correlations, and investigate whether items with negative correlations with the rest have been miscoded.
Look at the response format of the items. As it happens all the items in my survey have the 4-point format "Rarely, Not often, Often, Mostly". Presumably I would be happier using the mean as a summary measure if I had a 5+ point format, and if it used a more traditional Likert format with a neutral middle point, e.g. "Strongly disagree, Disagree, Neither agree nor disagree, Agree, Strongly agree".
Compare the mean score on the items with some other measure of the outcome, if I have one. I'm not sure what formal tests I'd do, but perhaps I could construct a Bland-Altman plot of the agreement between the measures.
Subjectively assess the items to check there aren't wild differences between them in terms of how related to the outcome they seem/how much they should be weighted. @Scortchi mentions a nice example here:
If your items are "I sometimes enjoy a salad for lunch", & "Meat is murder", what does the average score measure? Propensity to vegetarianism? Probably not: a score of 4 for the former & 2 for the latter will be a much weaker predictor than vice versa.