# Determining which scale items to use for further analysis

From a questionnaire survey I have 44 statements (items) which were supposed to measure 11 bases of power (factors). Four items per factor. Since this survey was only slightly altered from its well established original source, it is expected that eleven factors are actually measured. The number of observations is 258.

Due to the origin of the survey instrument and only slight adaptations made to the new context, I am unsure how to analyze the validity of the scales. Until now I have done the following:

• Grouped the descriptives of the 44 statements by the factor they should measure and looked for anomalies (describe() from the psych package).

• Created boxplots for the 11 bases of power - or rather their underlying scale items - and looked for anomalies (see here).

• Observed whether there is a drop in $\alpha$ when removing any of the scale items (alpha from the psych package).

• Observed whether there is an increase in average inter-item correlation when removing any of the scale items (alpha() from the psych package).

• Created an iclust plot for all 11 factors to see whether more items should be removed based on an increase of $\beta$ (iclust() from the psych package, see here).

By following this approach I end up removing five items due to an increase in $\alpha$, and one more item due to an increase in average inter-item correlation (and .01 decrease in $\alpha$). Some more items could be removed if I were strict with the increase in $\beta$.

Question 1: Should I remove items if this brings an increase in $\beta$ whilst the $\alpha$ remains unchanged?

Question 2: Should I remove items for a little increase in $\alpha$ or $\beta$ if the removal leads to less than three items for this factor?

Now having removed these items which are obviously not measure what they are supposed to, I tried to find a way to quantify the overall suitability of this measurement instrument. Since the number of factors is known and concluded from previous literature, I thought a confimatory factor analysis (CFA) would be a good idea (sem package). However no matter how hard I tried (parcelling based on the above iclust analysis, covariance between latent variables, modIndices() ) the best results were:

Model                   X²          DF      GFI     AGFI    RMSEA   NNFI    CFI     SRMR
38 items                1935.132    643     0.72    0.6773  0.0884  0.6678  0.6962  0.1661
Parcelling - 23 items   766.373     202     0.8168  0.7534  0.1032  0.7151  0.7691  0.1714

Now I wonder two things:

Question 3: Is it neccessary to report these findings?

Question 4: Should I ditch the idea of eleven factors and start over with exploratory factor analysis (EFA)?

I would face some challenges if the questionnaire hadn't measured what it was supposed to, hence the effort in keeping the eleven factors.

First of all I would say that you should definitely think about a latent variable approach. I don't know about the 11 bases of power you are talkng about. But it's definitely a psychological and as such a latent construct. This means that the questionaire cannot perfectly measure 11 distinguishable factors, even if it does indeed measure them. To account for this measurement error, you should do a confirmatory factor analysis, as you have already considered.

As far as your first two questions are concerned, this is also a better way to determine, if the items in a scale do measure the same construct. You should definitely not use Cronbach's $\alpha$, as it is not related to the internal structure of a questionaire. See the article of Sijtsma (2009) for a detailed account. This might come as a surprose to you - I know it certainly came as a surpise for me when I first learned about it. Especially since in the psychological literature, Cronbach's $\alpha$ is used all over the place.

If you do a confirmatory factor analysis, you can test different measurement models for your survey. It is relatively unlikely that a $\tau$-parallel model will fit the data. But an essential $\tau$-parallel model or a $\tau$-congeneric model might provide a decent or even a good fit. And there are a lot more possibilites to try. You can go a pretty long way, if you want.

Note that this does not have to do with the validity of the questionaire. Validity concerns the question, if the test is measuring what it is supposed to do. There are different ways to investigate validity. Usually you try to find other questionaires, that are supposed to measure something similar. Your questionaire should correlate highly with those. Likewise it should not correlate with questionaires that mesure something completely unrelated. For a very interesting critique of this approach see Borsboom et al. (2004).

As to your third question, yes, I would definitely report all your results, even if you wish they were different. This is also related to the fourth question. Maybe you have to give up the idea of 11 factors, because you just can't find support for it. I understand that that's not exactly what you wish for. But this would be a progress in knowledge as well. Coming from psychology myself, I know that there is a tendency to postulate some structure consisting of a number of - possibly even - independent factors. But very often, the assumptions just don't hold in empirical tests.

References:

Sijtsma, K. (2009). On the use, the misuse, and the very limited usefulness of Cronbach’s alpha. Psychometrika, 74(1), 107-120. LINK
Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2004). The concept of validity. Psychological review, 111(4), 1061. LINK

• +1 and thanks for the links. One thing though I can't help frowing at is "internal consistency", a vague term which I'd not recommend to use at all. For sometimes it is used for items-construct cohesion (validity) and sometimes for inter-items interchangeability (an aspect of reliability) – ttnphns Aug 14 '13 at 11:52
• @ttnphns I see your point and changed my wording. – Jens Kouros Aug 14 '13 at 12:06
• Thanks for the ideas and links Jens. I will look into $\tau$ congeneric models starting here, since I have no idea what this really means. The paper from Sijtsma (2009) was very enlightening about the usefulness of Cronbach's $\alpha$. I suppose for my purpose I better stay away of this measure. I'm very happy to accept your answer as a solution, as it opens me some more doors where to look at and which road to go down from here. Thanks. – Roman Aug 14 '13 at 12:11
• @Roman You're welcome. You should learn about Classical Test Theory, that's what this is about. From the website on your profile I get the impression that you speak german. Is that true? If so, I could refer you to this site which has excellent lectures on this topic. – Jens Kouros Aug 14 '13 at 12:22
• @Roman Every link on the site will lead to a whole semester worth of lectures. I know it's a lot, but they are very good. You can get a really good understanding of the theory that way. Also, you might not have to watch all of them. But there is a good chance you will want to. Unfortunately all the texts that I have read are from textbooks that are not freely available. I like this one and this one a lot. – Jens Kouros Aug 14 '13 at 12:45