13
$\begingroup$

I am running a structural equation model (SEM) in Amos 18. I was looking for 100 participants for my experiment (used loosely), which was deemed to be probably not enough to conduct successful SEM. I've been told repeatedly that SEM (along with EFA, CFA) is a "large sample" statistical procedure. Long story short, I didn't make it to 100 participants (what a surprise!), and only have 42 after excluding two problematic data points. Out of interest, I tried the model anyway, and to my surprise, it seemed to fit very well! CFI >.95, RMSEA < .09, SRMR <.08.

The model is not simple, in fact, I would say it is relatively complex. I have two latent variables, one with two observed and the other with 5 observed variables. I also have four additional observed variables in the model. There are numerous relationships between the variables, indirect and direct, with some variables being endogenous to four others, as an example.

I am somewhat new to SEM; however, two individuals that I know who are quite familiar with SEM tell me that as long as the fit indicies are good, the effects are interpretable (as long as they are significant) and there is nothing significantly "wrong" with the model. I know some fit indicies are biased for or against small samples in terms of suggesting good fit, but the three that I mentioned earlier seem fine, and I believe not similarly biased. To test for indirect effects I am using bootstrapping (2000 samples or so), 90 percent bias corrected confidence, monte carlo. An additional note is that I am running three different SEMs for three different conditions.

I have two questions that I would like some of you to consider and please reply to if you have something to contribute:

  1. Are there any significant weaknesses to my model that are not demonstrated by the fit indices? The small sample will be highlighted as a weakness of the study, but I am left wondering if there is some huge statistical problem that I am completely oblivious to. I plan on getting another 10-20 participants in the future, but this will still leave me with a relatively small sample for such analyses.

  2. Are there any problems with my use of bootstrapping given my small sample, or the context in which I am using it?

I hope these questions are not too "basic" for this forum. I have read a number of chapters on SEM and related matters, but I find people are very dispersed in terms of opinions in this area!

Cheers

$\endgroup$
  • 1
    $\begingroup$ @Behacad - well-explained problem. You are estimating a heck of a lot of parameters using very sparse data. So inference is going to be awfully shaky. But I'd like to step back and ask - are you using these 42 to infer relationships among a larger population? If so, is the 42 a random sample, or a least a demonstrably representative one? $\endgroup$ – rolando2 May 1 '11 at 23:02
  • $\begingroup$ Thank you for your comment rolando2! The sample includes 42 university students, and I am looking at the relationship between a number of factors and anxiety. The relationships I am looking to infer would be among the general population. My inferences are limited because participants are all relatively young students, but I am not looking for any specific population (e.g., individuals suffering from an anxiety disorder). I am interested in broadly stating for instance that X is indirectly associated with Y in a non-clinicam sample. Does that answer your questions? $\endgroup$ – Behacad May 2 '11 at 2:41
  • 1
    $\begingroup$ @Behacad - assuming you can defend to your potential critics the representativeness of your sample, I would say unequivocally that trying to estimate relationships among 12 variables is asking too much of your 42 cases. See if you can simplify your model to include just the 3 most interesting predictors. Although I realize it's painful to part with data you may have worked hard to collect! $\endgroup$ – rolando2 May 3 '11 at 3:00
  • $\begingroup$ Thank you for the reply. I have a "feeling" that estimating relationships amongst all these variables is hard to do with 42 data points, and I see where you are coming from. That being said, what would a statistical reason (preferably cited) for this problem be? How is this different from running a number of regressions/correlations on different dependent variables? The fit is good (and I'm actually running three different models for different experimental tasks) and results are consistent across models and in line with theory. Sorry if I'm coming off as defensive! $\endgroup$ – Behacad May 4 '11 at 15:54
  • $\begingroup$ (Not defensive--no worries!) Having 42 cases exposes you to sampling error at the very least, even when estimating univariate statistics. Now, in SEM, each variable gets used many times, because you're estimating the relationship of A to B while controlling for C, D, etc. So the effects of sampling error will be propagated, which, to my understanding, is why one typically wants large samples. In your case, you've got potentially more kinds of error than sampling error because you don't have a random sample. So you need to draw a very large credible interval around any result you obtain. $\endgroup$ – rolando2 May 4 '11 at 18:07
4
$\begingroup$

One point: there is no such thing as a "basic question", you only know what you know, and not what you don't know. asking a question is often the only way to find out.

Whenever you see small samples, you find out who really has "faith" in their models and who doesn't. I say this because small samples is usually where models have the biggest impact.

Being a keen (psycho?) modeller myself, I say go for it! You seem to be adopting a cautious approach, and you have acknowledged potential bias, etc. due to small sample. One thing to keep in mind with fitting models to small data is that you have 12 variables. Now you should think - how well could any model with 12 variables be determined by 42 observations? If you had 42 variables, then any model could be perfectly fit to those 42 observations (loosely speaking), so your case is not too far from being too flexible. What happens when your model is too flexible? It tends to fit the noise - that is, the relationships which are determined by things other than the ones you hypothesize.

You also have the opportunity to put your ego where your model is by predicting what those future 10-20 samples will be from your model. I wonder how your critics will react to a so called "dodgy" model which gives the right predictions. Note that you would get a similar "I told you so" if your model doesn't predict the data well.

Another way you could assure yourself that your results are reliable, is to try and break them. Keeping your original data intact, create a new data set, and see what you have to do to this new data set in order to make your SEM results seem ridiculous. Then look at what you had to do, and consider: is this a reasonable scenario? Does my "ridiculous" data resemble a genuine possibility? If you have to take your data to ridiculous territory in order to produce ridiculous results, it provides some assurance (heuristic, not formal) that your method is sound.

$\endgroup$
1
$\begingroup$

The main problem that I see with this is lack of power. Confirmatory factor and SEM testing look to accept the null - you want to see a non-significant p-value - so lack of power can be a problem. The power of the test depends on the sample size (42) and the degrees of freedom. AMOS gives you the degrees of freedom. You have not quoted it, but it won't be large in this case. With 12 variables, you start with 66 DF's, and subtract 1 for each parameter that you estimate. I don't know how many that would be, but you say that you have several factors and correlations between various constructs.

I do not entirely agree with Rolando2. In SEM, you gain by having lots of variables, assuming that they are reliable indicators of the underlying constructs. So don't reduce the number of variables. For the same reason, I do not entirely agree with @probabilityislogic. In SEM, you are not trying to model 12 variables with 42 observations. You are trying to model the constructs through 12 indicators, strengthened by 42 replications. A very simple factor model - 1 factor with 12 indicators - possibly could be tested with 42 people.

The RMSEA and other goodness of fit measures will tend to improve as you near saturation of the model, so again, you run the risk of a misleading result.

That being said, I have seen small data sets reject a factor model. It probably means something that the fit appears to be good.

Note: You can also check the residuals of a SEM model. These are the differences between the estimate covariance matrix and the model covariance matrix. AMOS will give them to you if you request them. Examination of the residuals might indicate if they are evenly distributed, or if certain covariances are very badly fitted.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.