Correlation between factors vs correlation between error terms of items measured by each factor

Question

Say there is a CFA model where there are 2+ factors, and the correlation between factors is freely estimated.

A change is made to the model such that an error term connected to an item on one factor is correlated with an error term connected to an item on another factor.

It’s been suggested to me that this will usually or always make the estimated correlation between the two factors increase.

Is that true? Or, on the contrary, will it usually or always make the estimated correlation between factors decrease. Why/why not?

Answer 1

It can go up, or down, or stay the same, depending on both the model and the data.

Here's a simple example. We have two factors, each has two indicators, with all loadings equal to 0.7 and a factor correlation of 0.5.

Create this data in R:

S <- matrix(rep(0.245, 16), nrow = 4)
diag(S) <- 1

S[1, 2] <- S[2, 1] <- S[3, 4] <- S[4, 3] <- 0.49

colnames(S) <- rownames(S) <- paste0("y", 1:4)

Fit the correct model:

library(lavaan) 
model_1 <- "
  f1 =~ y1 + y2
  f2 =~ y3 + y4
"

fit_1 <- lavaan::sem(model_1, sample.cov = S, sample.nobs = 1000)
summary(fit_1, standardized = TRUE)

Covariance between f1 and f2 is 0.245.

Add a correlation between y2 and y3 (which are indicators of different factors:

model_1a <- "
  f1 =~ y1 + y2
  f2 =~ y3 + y4
  y2 ~~ y3 
"

fit_1a <- lavaan::sem(model_1a, sample.cov = S, sample.nobs = 1000)
summary(fit_1a, standardized = TRUE)

The correlation and covariance between f1 and f2 is unchanged.

Now change the data. Increase the correlation of variables 2 and 3:

S[2, 3] <- S[3, 2] <- 0.64

Fit model 1 again:

fit_2 <- lavaan::sem(model_1, sample.cov = S, sample.nobs = 1000)
summary(fit_2, standardized = TRUE)

The covariance between f1 and f2 has dropped to 0.213, and the correlation has dropped to 0.435.

Add a correlated error:

fit_2a <- lavaan::sem(model_1a, sample.cov = S, sample.nobs = 1000)
summary(fit_2a, standardized = TRUE)

And the relationship between the latent variables is back to where it was. So adding the correlated error increases the covariance (and correlation) between the latent variables.

Try other values:

S <- matrix(rep(0.25, 16), nrow = 4)
diag(S) <- 1

S[2, 3] <- S[3, 2] <- 0.5

And the covariance / correlation goes down.

If there's non-zero residual correlation between two items on different scales, that correlation needs to go somewhere - so it changes the size of the correlation between the latent variables - which means that that estimate is biased, and the model is wrong without the correlated error and the estimates cannot be trusted.