2
$\begingroup$

EDIT: I have modified my data set so as to make it effectively decontextualized, and so now have data and example code if anyone wishes to reproduce this effect. You can find it at the bottom of the post.

I am currently running a path analysis on a set of data (I am using the "sem" function of the "lavaan" package in R).

The path diagram itself looks something like this:

"Opportunity-propensity" model

Where "Antecedent," "Opportunity," and "Propensity" refer to groups of observed variables (no latent factors are being used, for the moment) with paths between variables following the paths in the diagram (e.g. any variable in the "propensity" group is predicted by all variables in the "antecedent" group and the "opportunity" group, and predicts all variables in the "Pre-K Mathematics" group), and with "Pre-k Mathematics" being our primary research endpoint (it is not actually Pre-K mathematics in this study, but I could not find a suitably-simple picture with a generic endpoint).

(Note: being a lowly grad student, I do not get to choose the model we are using in this case; please do not respond with detailed questions regarding the motivation for this model, as I will not be able to answer them).

Now, I am observing something curious: the $R^2$ values for some of the endogenous variables (notably for our primary endpoint, but also for several others) are absolutely, unreasonably high (>.99). This seems to me an obvious red flag that something is going wrong..

Even more-interesting is that if I lump our "antecedent," "propensity," and "opportunity" variables into a single category and just do a simple linear regression for our endpoint, the $R^2$ drops to a much more reasonable (read: not particularly noteworthy) value of ~.15 (I strongly suspect there is not much of interest to see in this particular data set). Yet, the actual path coefficients for each variable to the endpoint do not change at all. This seems totally baffling - how can the variable's $R^2$ change drastically when the equation predicting it has remained the same?

It occurs to me that I must be missing something about how $R^2$ is calculated for endogenous variables in a SEM. Does anyone have any insight as to what might be causing this behavior? I am at a loss.

EDIT: As mentioned at the top, here is a sample data set which demonstrates the effect, along with a matching R script. The script reads the relevant files containing the data and model specifications (linked below), and outputs relevant summaries. Both the full path analysis and the simple linear regression are done using lavaan's "sem" method; for verification, the simple linear regression is also done using the "lm" method.

Data:

renamedData.csv

Model specifications:

renamedModel.txt renamedModelSimple.txt

Script:

library(lavaan)

renamedData <- read.csv("renamedData.csv")

renamedModel<-readLines("renamedmodel.txt")
renamedOut<-sem(renamedModel, renamedData)
summary(renamedOut, rsquare=T, standardized=T)

renamedModelSimple<-readLines("renamedmodelsimple.txt")
renamedOutSimple<-sem(renamedModelSimple, renamedData)
summary(renamedOutSimple, rsquare=T, standardized=T)

verification <- lm(data=renamedData, Out ~ A1 + A2 + O1 + O2 + O3 + O4 + O5 + O6 + O7 + P1 + P2 + P3 + P4 + P5)
summary(verification)

Note that I was not quite correct when I noted earlier that the path coefficients barely change at all - rather, they appear to be rigorously identical. I suppose that is somewhat suspicious, as well - but I have no idea how to interpret that, either.

$\endgroup$
  • $\begingroup$ Nicely written question. What do you mean by "single category"? How many predictors were used in that linear regression? $\endgroup$ – rolando2 Mar 21 '18 at 9:47
  • $\begingroup$ When you analyzed PreK Math as an outcome, was it the case that regression used only 1 of several highly correlated PreK Math variables while SEM used >1? $\endgroup$ – rolando2 Mar 21 '18 at 9:51
  • $\begingroup$ Consider disguising the data so that it can be posted, for example multiplying by the square root of two, or dividing by pi, so that the data is completely unusable except to help answer the question - sometimes this can be safely done. $\endgroup$ – James Phillips Mar 21 '18 at 13:23
  • $\begingroup$ @rolando2: By "single category," I mean simply running a multivariate linear regression of the "PreK math" variable versus all of the "antecedent," "opportunity," and "propensity" variables. Currently, there are 3 "antecedent" variables, 6 "opportunity" variables, and 5 "propensity" variables; so, in the simple linear regression, there were 14 predictors. (I am aware of the potential overfitting problems induced by having many predictors; I intend to use 10-fold cross-validation or similar at some point in the future to help compensate for this). $\endgroup$ – user3716267 Mar 21 '18 at 14:17
  • $\begingroup$ @rolando2: It was not the case that the simple regression used only 1 of several highly-correlated variables, while the SEM used more than 1. The coefficients for each predictor (note that the set of predictors is the same in both the SEM and the simple regression) were very nearly identical, which is particularly baffling to me. $\endgroup$ – user3716267 Mar 21 '18 at 14:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.