Determining which scale items to use for further analysis

Question

From a questionnaire survey I have 44 statements (items) which were supposed to measure 11 bases of power (factors). Four items per factor. Since this survey was only slightly altered from its well established original source, it is expected that eleven factors are actually measured. The number of observations is 258.

Due to the origin of the survey instrument and only slight adaptations made to the new context, I am unsure how to analyze the validity of the scales. Until now I have done the following:

• Grouped the descriptives of the 44 statements by the factor they should measure and looked for anomalies (describe() from the psych package).

• Created boxplots for the 11 bases of power - or rather their underlying scale items - and looked for anomalies (see here).

• Observed whether there is a drop in $\alpha$ when removing any of the scale items (alpha from the psych package).

• Observed whether there is an increase in average inter-item correlation when removing any of the scale items (alpha() from the psych package).

• Created an iclust plot for all 11 factors to see whether more items should be removed based on an increase of $\beta$ (iclust() from the psych package, see here).

By following this approach I end up removing five items due to an increase in $\alpha$, and one more item due to an increase in average inter-item correlation (and .01 decrease in $\alpha$). Some more items could be removed if I were strict with the increase in $\beta$.

Question 1: Should I remove items if this brings an increase in $\beta$ whilst the $\alpha$ remains unchanged?

Question 2: Should I remove items for a little increase in $\alpha$ or $\beta$ if the removal leads to less than three items for this factor?

Now having removed these items which are obviously not measure what they are supposed to, I tried to find a way to quantify the overall suitability of this measurement instrument. Since the number of factors is known and concluded from previous literature, I thought a confimatory factor analysis (CFA) would be a good idea (sem package). However no matter how hard I tried (parcelling based on the above iclust analysis, covariance between latent variables, modIndices() ) the best results were:

Model                   X²          DF      GFI     AGFI    RMSEA   NNFI    CFI     SRMR 
38 items                1935.132    643     0.72    0.6773  0.0884  0.6678  0.6962  0.1661 
Parcelling - 23 items   766.373     202     0.8168  0.7534  0.1032  0.7151  0.7691  0.1714

Now I wonder two things:

Question 3: Is it neccessary to report these findings?

Question 4: Should I ditch the idea of eleven factors and start over with exploratory factor analysis (EFA)?

I would face some challenges if the questionnaire hadn't measured what it was supposed to, hence the effort in keeping the eleven factors.

Answer 1

First of all I would say that you should definitely think about a latent variable approach. I don't know about the 11 bases of power you are talkng about. But it's definitely a psychological and as such a latent construct. This means that the questionaire cannot perfectly measure 11 distinguishable factors, even if it does indeed measure them. To account for this measurement error, you should do a confirmatory factor analysis, as you have already considered.

As far as your first two questions are concerned, this is also a better way to determine, if the items in a scale do measure the same construct. You should definitely not use Cronbach's $\alpha$, as it is not related to the internal structure of a questionaire. See the article of Sijtsma (2009) for a detailed account. This might come as a surprose to you - I know it certainly came as a surpise for me when I first learned about it. Especially since in the psychological literature, Cronbach's $\alpha$ is used all over the place.

If you do a confirmatory factor analysis, you can test different measurement models for your survey. It is relatively unlikely that a $\tau$-parallel model will fit the data. But an essential $\tau$-parallel model or a $\tau$-congeneric model might provide a decent or even a good fit. And there are a lot more possibilites to try. You can go a pretty long way, if you want.

Note that this does not have to do with the validity of the questionaire. Validity concerns the question, if the test is measuring what it is supposed to do. There are different ways to investigate validity. Usually you try to find other questionaires, that are supposed to measure something similar. Your questionaire should correlate highly with those. Likewise it should not correlate with questionaires that mesure something completely unrelated. For a very interesting critique of this approach see Borsboom et al. (2004).

As to your third question, yes, I would definitely report all your results, even if you wish they were different. This is also related to the fourth question. Maybe you have to give up the idea of 11 factors, because you just can't find support for it. I understand that that's not exactly what you wish for. But this would be a progress in knowledge as well. Coming from psychology myself, I know that there is a tendency to postulate some structure consisting of a number of - possibly even - independent factors. But very often, the assumptions just don't hold in empirical tests.

References:

_{Sijtsma, K. (2009). On the use, the misuse, and the very limited usefulness of Cronbach’s alpha. Psychometrika, 74(1), 107-120. LINK
Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2004). The concept of validity. Psychological review, 111(4), 1061. LINK}