Which package to use for convergent and discriminant validity in R?

Question

I am new to R and want to assess validity of some scales using R. Is there a package that I can use to test for convergent and discriminant validity in R?

Answer 1

The most commonly used correlational approach is called multi-trait scaling (MTS). The idea behind MTS is that items belonging to the same dimension are supposed to be more correlated to each other, as well as to their own scale, and that their correlations with items from other dimensions should be zero or at least of a lower magnitude. We generally consider that a correlation of 0.3 (or 0.4) or above is indicative of a 'meaningful' correlation for two items belonging to the same subscale. Scaling success is the proportion of items having a larger correlation with their own dimension than with any other dimensions, with higher values indicating good convergent validity. Basically, we compute interitem correlations within each scale and between scales; that is, for each subscale, every item score is correlated with every other item from the same subscale (convergent validity) and from other scales (divergent validity). Such a method is readily available in the psy as mtmm(); it gives numerical results and graphical output when requested. Some authors have suggested to examine one-tailed correlation tests, with appropriate correction for multiple comparisons (most of the times, a conservative approach like Bonferroni method). Scaling success with significance tests refers to the number of significant correlations (i.e., only non-significant one-tailed correlation tests are counted as scaling errors).

Finally, one generally report the average homogeneity index (within-scale interitem correlations) and an indicator of internal consistency like Cronbach's alpha. The above terminology comes from Fayers and Machin (2007). The psy::mtmm() function does not perform significance tests, but it is not really hard to code.

An example of a summary table for measures of convergent and divergent validity is shown below:

(It comes from the SF-36 questionnaire that is used to measure health-related quality of life. Table 5.2, p. 119, in (1).)

You will notice that I called such method MTS although it comes along Multitrait Multimethod (MTMM) analysis (2) which has been discussed on a related question, How to compute correlation between/within groups of variables? That's not surprising as the latter is just a generalization of MTS applied to multiple instrument: In MTMM we not only focus on the way a given instrument measures one or more traits but compare it to other known instruments, also called methods. There are various formulations of MTMM models, including correlated uniqueness model (3), CFA model for MTMM (4,5), the direct product model (6), and the true score (TS) model (7). Geometric representations from CFA models are discussed in (8). As can be seen, those approaches are not based on simple correlation analysis but allow to postulate a real measurement model (see next paragraphs on factor analysis), although it has been the subject of some criticisms, e.g. (9-11).

Another idea is to compute correlation between item and subscale scores (e.g., bi- or polyserial correlation), for each subscale (this can be done with other subscales as well). High correlations are indicative of a good discriminatory power and altogether they will support the idea that items belonging to the same subscale are correlated one to each other and to the total score (as reflected by Cronbach's alpha, loading in factor analysis, or the discrimination parameter in an item response model).

Another approach relies on factor analytical methods, either exploratory or confirmatory factor analysis (CFA). In the CFA framework, the pattern matrix (correlations between items and factor, or loadings) of the hypothesized measurement model is specified in advance, whereas in exploratory factor analysis loadings, communalities and uniquenesses are estimated from the data without imposing constraints. Two critical points that are worth to remember are that (a) if the same sample is used to extract the factors (and construct scoring rules for each subscale) and assess goodness of fit of a factor model, then the generalizability of CFA findings will obviously be limited, and (b) if the model does not fit the data, nothing tell us what could be the correct model (even if so-called modification indices are used to refine the initial model, because in this case it is easy to fall into the trap of data snooping). In R, there are three packages that might be used: sem, lavaan, OpenMx. All three packages provide numerous example of CFA applications.

The following are rough guidelines for assessing model fit: a Comparative Fit (CFI) and Tucker-Lewis Index (TLI) greater than .90 are indicative of adequate model fit, with values near .95 being preferable; a Standardised Root Mean Square Residual (SRMR) below .10 (resp. .08) and a Root Mean Square Error of Approximation (RMSEA) below .08 (resp. .06) are indicative of acceptable (resp. good) model fit (12).

As a sidenote, I would recommend having a look at the psych package as well. It features a lot of useful methods for psychometrics (item and reliability analysis, factor-related methods). William Revelle has a book in progress on applied psychometrics with R.

References

1. Fayers, P. M. and Machin, D. (2007). Quality of Life: The assessment, analysis and interpretation of patient-reported outcomes. Wiley.
2. Campbell, D.T. and Fiske, D.W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105.
3. Marsh, H.W. (1989). Confirmatory factor analysis of multitrait-multimethod data: Many problems and few solutions. Applied Psychological Measurement, 13, 335–361.
4. Althauser, R.P., Heberlein, T.A., and Scott, R.A. (1971). A causal assessment of validity: The augmented multitrait-multimethod matrix. In Blalock, H.M. (editor), Causal Models in the Social Sciences, pp. 151–169. Chicago: Aldine.
5. Andrews, F.M. (1984). Construct validity and error components of survey measures: a structural modeling approach. Public Opinion Quarterly, 48, 409–442.
6. Browne, M.W. (1984). The decomposition of multitrait-multimethod matrices. British Journal of Mathematical and Statistical Psychology, 37, 1–21.
7. Saris, W.E and Andrews, F.M. (1991). Evaluation of measurement instruments using a structural modeling approach. In Biemer, P.P., Groves, R.M., Lyberg, L.E., Mathiowetz, N.A., and Sudman, S. (editors), Measurement Errors in Surveys, pp. 575–597. New York: Wiley.
8. Hox, J.J. and Bechger, T. (1995). Comparing and combining different approaches to the Multitrait-Multimethod matrix. In Hox, J.J., Mellenbergh, G.J., and Swanborn, P.G. (editors), Facet Theory: Analysis and Design, pp. 135-144. Zeist, the Netherlands: SETOS.
9. Brannick, M.T and Spector, P.E. (1990). Estimation Problems in the Block-Diagonal Model of the Multitrait-Multimethod Matrix. Applied Psychological Measurement, 14(4), 325-339.
10. Putka, D.J., Lance, C.E., Le, H., and McCloy, R.A. (2011). A Cautionary Note on Modeling Multitrait-Multirater Data Arising From Ill-Structured Measurement Designs. Organizational Research Methods, 14(3), 503-529.
11. Cote, J.A. (1995), What causes estimation problems when analyzing MTMM data? In Kardes, F.R. and Sujan, M. (editors), Advances in Consumer Research Volume 22, pp. 345-353. Provo, UT: Association for Consumer Research.
12. Hu, L. T. and Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-55.