I'm working on a questionnaire that is intended to measure two constructs. There are presently 26 items in the questionnaire -- eleven for construct 1 and 15 for construct 2 -- and each item has five available responses: "strongly disagree" "disagree" "neutral" "agree" "strongly agree".
In our first iteration of the questionnaire, we used ten items for each construct, and had about 60 respondents. After the results came in, we made some judgments based on item discrimination (dichotomized), point-biserial correlation, cronbach's alpha after item deletion, and factor analysis, and dropped several of these questions and replaced them with others.
We now have 68 responses to the present questionnaire (the one with 26 items). I'd like to make sure I cover my bases as we do the reliability and validity analysis, and am specifically wondering if we should carry out additional diagnoses or approach the ones we've already calculated in different ways. Our approach to identifying potentially problematic items is to look for:
item discriminations below .3 on the dichotomized responses - where agree/strongly is coded as 1 and the others are coded as zero, and we take the difference between the top 3'rd of respondents and the bottom 3'rd of respondents.
low point-biserial (spearman's) correlations between items and the total score for the associated construct (total score = mean response for that construct). We don't have a threshold defining "low" ... and is spearman's or pearson's the appropriate measure?
items whose inclusion causes cronbach's alpha to decrease -- identified by running tests with "cronbach's alpha after item is deleted." Would G6 or Omega be more appropriate here?
Factor analysis -- we've been looking for items with low loadings in either dimension. Here, we're wondering about two thngs: a) whether to "swap" factors from their intended construct to the one which they appear to load with more strongly. Is that a defensible approach, even if the measure doesn't have face validity with respect to its intended construct? b) if an item loads strongly in both dimensions (where it appears that there's little separation) is that a problem? I didn't think it was but I thought it might be worthwhile to add.
If anyone has any thoughts on these questions, or if an important procedure comes to mind that we haven't consider, I'd love to hear it. Thanks in advance, and I apologize if this post is not properly directed!