1
$\begingroup$

I built a causal model for investigating help-seeking. I have collected data to test my model through SEM. I have collected data three times from the same population over three months (using the same questionnaire).

I was able to obtain a good fit for each collected data set. However, for example, one of the path (relationship) between concept A and B is not significant for the first data set, but this relationship is significant for the other two data sets. However, many of the relationships are consistently significant for three data sets.

I can exclude the non-significant (NS) paths and re-run SEM and obtain a better model fit. However, I guess I should keep these NS paths and discuss why they are not significant, and highlight the ones that are consistently significant.

What do you think about this method? If you were able to test a causal model with three different data sets, would you discard the NS paths? How would you organize your results, and interpret them?

Best!

Here is my Mplus code:

VARIABLE:

  NAMES = X1-X41;!X59;
  USEVARIABLES ARE X1-X4 X5 X7 X10 X12-X41; !x1=id x2=g x3=t
  GROUPING IS X3 (1 = 1 2 = 2 3 = 3);
  cluster is X1;
ANALYSIS:
  TYPE IS complex;
  ESTIMATOR=ML;

model:

  REL BY X4 X5 X7 X10;
  TS BY X12-X15;
  INT BY X16-X18;
  COSTS BY X19-X22;
  BENFS BY X23-X25;
  INST BY X26-X28;
  EXEC BY X29-X31;
  MAST BY X32-X35; 
  PERM BY X36-X41;

model 1:

  BENFS ON TS REL MAST (a1);
  COSTS ON TS REL PERM (a2);

  EXEC ON PERM COSTS(a3);
  INST ON MAST BENFS(a4);

  INT ON BENFS COSTS MAST REL(a5);


model 2:
  BENFS ON TS REL MAST (b1);
  COSTS ON TS REL PERM (b2);

  EXEC ON PERM COSTS(b3);
  INST ON MAST BENFS(b4);

  INT ON BENFS COSTS MAST REL(b5);

model 3:
  BENFS ON TS REL MAST (c1);
  COSTS ON TS REL PERM (c2);

  EXEC ON PERM COSTS(c3);
  INST ON MAST BENFS(c4);

  INT ON BENFS COSTS MAST REL(c5);


model constraint:
 a1 = b1;
 a1 = c1;

 a2 = b2;
 a2 = c2;

 a3 = b3;
 a3 = c3;

 a4 = b4;
 a4 = c4;

 a5 = b5;
 a5 = c5;
$\endgroup$
6
  • 2
    $\begingroup$ This is hard to follow. Can you make this more concrete? Can you provide a simplified example, eg? $\endgroup$ Oct 3 '15 at 23:47
  • $\begingroup$ @gung I tried to reorganize and simplify my writing, does it make sense now? $\endgroup$
    – renakre
    Oct 4 '15 at 0:00
  • $\begingroup$ If you want to include longitudinal effects then you can set up a longitudinal model. You could also fit a single model using a multilevel SEM. $\endgroup$ Oct 6 '15 at 16:31
  • $\begingroup$ @JeremyMiles with multiple datasets my purpose is to be able to more strongly argue that the model is valid one, I am not interested in longitudinal effects. $\endgroup$
    – renakre
    Oct 6 '15 at 23:51
  • $\begingroup$ In that case I'd set it up as a clustered / multilevel model, with the datasets appended. You can then estimate the parameters in all three models simultaneously. What software do you use? $\endgroup$ Oct 6 '15 at 23:52
1
+50
$\begingroup$

We can think of what you have as a study that has been replicated three times. The problem with doing the study three times is that it is hard to distinguish between random variation and real differences. If these were different people we could combine the three studies in something like an Individual Patient Data meta-analysis (IPDA). We can do something like that in Mplus, using multiple groups.

We stack the data so that it is tall, and we have a time identifier. I've put a really simple example, with one variable, on Dropbox, here: https://db.tt/GSdAUCTt

We have three variables: id, which goes from 1 to 100, and because we replicated the study three times, each person appears three times. t, which is time - it's 1 for the first time, 2 for the second, etc. Then a single variable, which I've called x.

Here's my variable set up in Mplus:

  VARIABLE:
    NAMES ARE id t x;
    USEVARIABLES ARE id t x;
    GROUPING IS t (1 = 1 2 = 2 3 = 3);

The grouping tells Mplus that I have three groups, called 1, 2 and 3. However, these groups aren't indepdendent - and I need to tell MPlus that, using the cluster option.

cluster is id;

Then we fit a multiple group model in Mplus. This means that we estimate each parameter in each group. In my model, I'm just going to estimate the mean and variance of x in each group. Your model will be more complex, but the results will be exactly the same as if you estimated the three groups separately.

  model:
  model 1:
  x (var1);
  [x] (mean1) ;

  model 2:
  x (var2);
  [x] (mean2) ;

  model 3:
  x (var3);
  [x] (mean3) ;

Now I've estimated my three groups. But I think that all the parameters should be the same. I labelled each parameter, so now I constrain them to be equal, and I'll estimate a single parameter across the three groups. To do that, I add:

 model constraint:
 var1 = var2;
 var1 = var3;

 mean1 = mean2;
 mean1 = mean3;

Checking my model results, I have the same values, standard errors, etc, in all three groups.

                                  Two-Tailed
                    Estimate       S.E.  Est./S.E.    P-Value

Group 1

 Means
    X                  4.873      0.166     29.303      0.000

 Variances
    X                  8.174      0.447     18.296      0.000

Group 2

 Means
    X                  4.873      0.166     29.303      0.000

 Variances
    X                  8.174      0.447     18.296      0.000

Group 3

 Means
    X                  4.873      0.166     29.303      0.000

 Variances
    X                  8.174      0.447     18.296      0.000

How do I know that it was OK to combine them? I can do a chi-square difference test, and test if my null hypothesis that the parameters don't differ across the three groups. That's done with the chi-square difference test of the first model against the second. (Note that it's an MLR estimator, so you need to do the MLR difference test - which is described here: https://www.statmodel.com/chidiff.shtml There are excel spreadsheets floating around the internet to do it for you.

My Mplus file can also be found here: https://db.tt/aXjeLLDc

It might be the case that there is some sort of learning effect, and that you can't constrain the means to be equal across groups. If that happens, I don't think it's a problem.

Based on the comment below, you can name constraints for more elaborate models. The variables have the same name - it's the group that identifies which study it is from

model 1:
a on b (ab1);
a on c (ac1);
a on d (ad1);

model 2:
a on b (ab2);
a on c (ac2);
a on d (ad2);

model 3:
a on b (ab3);
a on c (ac3);
a on d (ad3);

model constraint: 
ab1 = ab2;
ab1 = ab3;

ac1 = ac2;
ac1 = ac3;

ad1 = ad2;
ad1 = ad3;
$\endgroup$
25
  • $\begingroup$ is the main idea behind your design to test if these three models are significantly different from each other or not? Is this why you constraint them to be the same? So, at the end basically I will have a single model results since they are all equalized? $\endgroup$
    – renakre
    Oct 7 '15 at 12:27
  • $\begingroup$ also, what would be the difference if I ONLY do a chi-square difference among three models and try to see which model fits the best, and just use it? I guess you combined all three sets for a single model? $\endgroup$
    – renakre
    Oct 7 '15 at 14:56
  • $\begingroup$ The purpose of the chi-square test is to compare the three models and see if you can find differences. If you can't find differences, then you merge the three into one by constraining everything to be equal. There are two advantages of this: First, you have one model, not three, so you have an easier time. Second, your parameters are all estimated using all of the data, so should have smaller standard errors and you have more power to detect your effects. $\endgroup$ Oct 7 '15 at 15:58
  • 1
    $\begingroup$ :) That's OK. Differences are fine. If everyone was different, then it's the same as not having the cluster line. $\endgroup$ Oct 9 '15 at 19:01
  • 1
    $\begingroup$ Well done. If the constraints are tenable (via chi-square test) report the constrained. Otherwise unconstrained. $\endgroup$ Feb 21 '16 at 5:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.