From a questionnaire survey I have 44 statements (items) which were supposed to measure 11 bases of power (factors). Four items per factor. Since this survey was only slightly altered from its well established original source, it is expected that eleven factors are actually measured. The number of observations is 258.
Due to the origin of the survey instrument and only slight adaptations made to the new context, I am unsure how to analyze the validity of the scales. Until now I have done the following:
Grouped the descriptives of the 44 statements by the factor they should measure and looked for anomalies (
describe()from the psych package).
Created boxplots for the 11 bases of power - or rather their underlying scale items - and looked for anomalies (see here).
Observed whether there is a drop in $\alpha$ when removing any of the scale items (
alphafrom the psych package).
Observed whether there is an increase in average inter-item correlation when removing any of the scale items (
alpha()from the psych package).
iclustplot for all 11 factors to see whether more items should be removed based on an increase of $\beta$ (
iclust()from the psych package, see here).
By following this approach I end up removing five items due to an increase in $\alpha$, and one more item due to an increase in average inter-item correlation (and .01 decrease in $\alpha$). Some more items could be removed if I were strict with the increase in $\beta$.
Question 1: Should I remove items if this brings an increase in $\beta$ whilst the $\alpha$ remains unchanged?
Question 2: Should I remove items for a little increase in $\alpha$ or $\beta$ if the removal leads to less than three items for this factor?
Now having removed these items which are obviously not measure what they are supposed to, I tried to find a way to quantify the overall suitability of this measurement instrument. Since the number of factors is known and concluded from previous literature, I thought a confimatory factor analysis (CFA) would be a good idea (
sem package). However no matter how hard I tried (parcelling based on the above
iclust analysis, covariance between latent variables,
modIndices() ) the best results were:
Model X² DF GFI AGFI RMSEA NNFI CFI SRMR 38 items 1935.132 643 0.72 0.6773 0.0884 0.6678 0.6962 0.1661 Parcelling - 23 items 766.373 202 0.8168 0.7534 0.1032 0.7151 0.7691 0.1714
Now I wonder two things:
Question 3: Is it neccessary to report these findings?
Question 4: Should I ditch the idea of eleven factors and start over with exploratory factor analysis (EFA)?
I would face some challenges if the questionnaire hadn't measured what it was supposed to, hence the effort in keeping the eleven factors.