I have dataset of test items which were administered in blocks. As a result, not all students answered each test item and there are some pairs of items for which no observations are shared, introducing missing values into the covariance matrix. As an example, suppose I have the following data. I'll provide an example in Python but your answer doesn't need to depend on a programming language.
import numpy as np import pandas as pd random.seed(23) cov = np.array([[2,0.5],[0.5,1]]) s = np.random.multivariate_normal(mean=[0, 0], cov=[[2,0.5],[0.5,1]], size=N) m1 = 0.5 + 1*s[:, 0] + np.random.normal(scale=2, size=N) m2 = 2 + 1*s[:, 1] + np.random.normal(scale=1.5, size=N) m3 = -3 + 2*s[:, 0] -4*s[:, 1] + np.random.normal(scale=4, size=N) m4 = -1 + 1.5*s[:, 0] + 2*s[:, 1] + np.random.normal(scale=0.5, size=N) m5 = 3 - 1*s[:, 0] + 1*s[:, 1] + np.random.normal(scale=3, size=N) m6 = 2 + 1.2*s[:, 0] + 5*s[:, 1] + np.random.normal(scale=1, size=N) M = np.vstack([m1, m2, m3, m4, m5, m6]).T M[:int(N/2), 4] = np.nan M[int(N/2):, 5] = np.nan print(pd.DataFrame(M).corr().values)
The resulting covariance matrix is
array([[ 1. , 0.12, 0.16, 0.47, -0.19, 0.32], [ 0.12, 1. , -0.3 , 0.45, 0.06, 0.53], [ 0.16, -0.3 , 1. , -0.16, -0.27, -0.39], [ 0.47, 0.45, -0.16, 1. , -0.12, 0.91], [-0.19, 0.06, -0.27, -0.12, 1. , nan], [ 0.32, 0.53, -0.39, 0.91, nan, 1. ]])
which shows that I have missing values for the covariance on the last two measures. In my exact problem I have additional missing components.
I'd like to estimate a confirmatory factor analysis model using all of the available data, but I'm not sure how to do it with these missing values. I will be fixing a priori zeros (see Rubin and Thayer, 1982) in the loading matrix to secure identification, so I need the flexibility to be able to do this.
I'd be open to learning if this is possible in
factanal in R or using packages in
Python. I'd also like to know the references for the methods used to address this problem. I'm thinking there must be some FIML approach. But most articles I've found address only the case where the data contain missing values but there at least a few observations that have nonmissing values for each pair of columns.