how to deal with three sets of SEM results on the same causal model

Question

I built a causal model for investigating help-seeking. I have collected data to test my model through SEM. I have collected data three times from the same population over three months (using the same questionnaire).

I was able to obtain a good fit for each collected data set. However, for example, one of the path (relationship) between concept A and B is not significant for the first data set, but this relationship is significant for the other two data sets. However, many of the relationships are consistently significant for three data sets.

I can exclude the non-significant (NS) paths and re-run SEM and obtain a better model fit. However, I guess I should keep these NS paths and discuss why they are not significant, and highlight the ones that are consistently significant.

What do you think about this method? If you were able to test a causal model with three different data sets, would you discard the NS paths? How would you organize your results, and interpret them?

Best!

Here is my Mplus code:

VARIABLE:

  NAMES = X1-X41;!X59;
  USEVARIABLES ARE X1-X4 X5 X7 X10 X12-X41; !x1=id x2=g x3=t
  GROUPING IS X3 (1 = 1 2 = 2 3 = 3);
  cluster is X1;
ANALYSIS:
  TYPE IS complex;
  ESTIMATOR=ML;

model:

  REL BY X4 X5 X7 X10;
  TS BY X12-X15;
  INT BY X16-X18;
  COSTS BY X19-X22;
  BENFS BY X23-X25;
  INST BY X26-X28;
  EXEC BY X29-X31;
  MAST BY X32-X35; 
  PERM BY X36-X41;

model 1:

  BENFS ON TS REL MAST (a1);
  COSTS ON TS REL PERM (a2);

  EXEC ON PERM COSTS(a3);
  INST ON MAST BENFS(a4);

  INT ON BENFS COSTS MAST REL(a5);


model 2:
  BENFS ON TS REL MAST (b1);
  COSTS ON TS REL PERM (b2);

  EXEC ON PERM COSTS(b3);
  INST ON MAST BENFS(b4);

  INT ON BENFS COSTS MAST REL(b5);

model 3:
  BENFS ON TS REL MAST (c1);
  COSTS ON TS REL PERM (c2);

  EXEC ON PERM COSTS(c3);
  INST ON MAST BENFS(c4);

  INT ON BENFS COSTS MAST REL(c5);


model constraint:
 a1 = b1;
 a1 = c1;

 a2 = b2;
 a2 = c2;

 a3 = b3;
 a3 = c3;

 a4 = b4;
 a4 = c4;

 a5 = b5;
 a5 = c5;

Answer 1

We can think of what you have as a study that has been replicated three times. The problem with doing the study three times is that it is hard to distinguish between random variation and real differences. If these were different people we could combine the three studies in something like an Individual Patient Data meta-analysis (IPDA). We can do something like that in Mplus, using multiple groups.

We stack the data so that it is tall, and we have a time identifier. I've put a really simple example, with one variable, on Dropbox, here: https://db.tt/GSdAUCTt

We have three variables: id, which goes from 1 to 100, and because we replicated the study three times, each person appears three times. t, which is time - it's 1 for the first time, 2 for the second, etc. Then a single variable, which I've called x.

Here's my variable set up in Mplus:

  VARIABLE:
    NAMES ARE id t x;
    USEVARIABLES ARE id t x;
    GROUPING IS t (1 = 1 2 = 2 3 = 3);

The grouping tells Mplus that I have three groups, called 1, 2 and 3. However, these groups aren't indepdendent - and I need to tell MPlus that, using the cluster option.

cluster is id;

Then we fit a multiple group model in Mplus. This means that we estimate each parameter in each group. In my model, I'm just going to estimate the mean and variance of x in each group. Your model will be more complex, but the results will be exactly the same as if you estimated the three groups separately.

  model:
  model 1:
  x (var1);
  [x] (mean1) ;

  model 2:
  x (var2);
  [x] (mean2) ;

  model 3:
  x (var3);
  [x] (mean3) ;

Now I've estimated my three groups. But I think that all the parameters should be the same. I labelled each parameter, so now I constrain them to be equal, and I'll estimate a single parameter across the three groups. To do that, I add:

 model constraint:
 var1 = var2;
 var1 = var3;

 mean1 = mean2;
 mean1 = mean3;

Checking my model results, I have the same values, standard errors, etc, in all three groups.

                                  Two-Tailed
                    Estimate       S.E.  Est./S.E.    P-Value

Group 1

 Means
    X                  4.873      0.166     29.303      0.000

 Variances
    X                  8.174      0.447     18.296      0.000

Group 2

 Means
    X                  4.873      0.166     29.303      0.000

 Variances
    X                  8.174      0.447     18.296      0.000

Group 3

 Means
    X                  4.873      0.166     29.303      0.000

 Variances
    X                  8.174      0.447     18.296      0.000

How do I know that it was OK to combine them? I can do a chi-square difference test, and test if my null hypothesis that the parameters don't differ across the three groups. That's done with the chi-square difference test of the first model against the second. (Note that it's an MLR estimator, so you need to do the MLR difference test - which is described here: https://www.statmodel.com/chidiff.shtml There are excel spreadsheets floating around the internet to do it for you.

My Mplus file can also be found here: https://db.tt/aXjeLLDc

It might be the case that there is some sort of learning effect, and that you can't constrain the means to be equal across groups. If that happens, I don't think it's a problem.

Based on the comment below, you can name constraints for more elaborate models. The variables have the same name - it's the group that identifies which study it is from

model 1:
a on b (ab1);
a on c (ac1);
a on d (ad1);

model 2:
a on b (ab2);
a on c (ac2);
a on d (ad2);

model 3:
a on b (ab3);
a on c (ac3);
a on d (ad3);

model constraint: 
ab1 = ab2;
ab1 = ab3;

ac1 = ac2;
ac1 = ac3;

ad1 = ad2;
ad1 = ad3;