Complications of having a very small sample in a structural equation model

Question

I am running a structural equation model (SEM) in Amos 18. I was looking for 100 participants for my experiment (used loosely), which was deemed to be probably not enough to conduct successful SEM. I've been told repeatedly that SEM (along with EFA, CFA) is a "large sample" statistical procedure. Long story short, I didn't make it to 100 participants (what a surprise!), and only have 42 after excluding two problematic data points. Out of interest, I tried the model anyway, and to my surprise, it seemed to fit very well! CFI >.95, RMSEA < .09, SRMR <.08.

The model is not simple, in fact, I would say it is relatively complex. I have two latent variables, one with two observed and the other with 5 observed variables. I also have four additional observed variables in the model. There are numerous relationships between the variables, indirect and direct, with some variables being endogenous to four others, as an example.

I am somewhat new to SEM; however, two individuals that I know who are quite familiar with SEM tell me that as long as the fit indicies are good, the effects are interpretable (as long as they are significant) and there is nothing significantly "wrong" with the model. I know some fit indicies are biased for or against small samples in terms of suggesting good fit, but the three that I mentioned earlier seem fine, and I believe not similarly biased. To test for indirect effects I am using bootstrapping (2000 samples or so), 90 percent bias corrected confidence, monte carlo. An additional note is that I am running three different SEMs for three different conditions.

I have two questions that I would like some of you to consider and please reply to if you have something to contribute:

1. Are there any significant weaknesses to my model that are not demonstrated by the fit indices? The small sample will be highlighted as a weakness of the study, but I am left wondering if there is some huge statistical problem that I am completely oblivious to. I plan on getting another 10-20 participants in the future, but this will still leave me with a relatively small sample for such analyses.

2. Are there any problems with my use of bootstrapping given my small sample, or the context in which I am using it?

I hope these questions are not too "basic" for this forum. I have read a number of chapters on SEM and related matters, but I find people are very dispersed in terms of opinions in this area!

Cheers

Answer 1

One point: there is no such thing as a "basic question", you only know what you know, and not what you don't know. asking a question is often the only way to find out.

Whenever you see small samples, you find out who really has "faith" in their models and who doesn't. I say this because small samples is usually where models have the biggest impact.

Being a keen (psycho?) modeller myself, I say go for it! You seem to be adopting a cautious approach, and you have acknowledged potential bias, etc. due to small sample. One thing to keep in mind with fitting models to small data is that you have 12 variables. Now you should think - how well could any model with 12 variables be determined by 42 observations? If you had 42 variables, then any model could be perfectly fit to those 42 observations (loosely speaking), so your case is not too far from being too flexible. What happens when your model is too flexible? It tends to fit the noise - that is, the relationships which are determined by things other than the ones you hypothesize.

You also have the opportunity to put your ego where your model is by predicting what those future 10-20 samples will be from your model. I wonder how your critics will react to a so called "dodgy" model which gives the right predictions. Note that you would get a similar "I told you so" if your model doesn't predict the data well.

Another way you could assure yourself that your results are reliable, is to try and break them. Keeping your original data intact, create a new data set, and see what you have to do to this new data set in order to make your SEM results seem ridiculous. Then look at what you had to do, and consider: is this a reasonable scenario? Does my "ridiculous" data resemble a genuine possibility? If you have to take your data to ridiculous territory in order to produce ridiculous results, it provides some assurance (heuristic, not formal) that your method is sound.

Answer 2

The main problem that I see with this is lack of power. Confirmatory factor and SEM testing look to accept the null - you want to see a non-significant p-value - so lack of power can be a problem. The power of the test depends on the sample size (42) and the degrees of freedom. AMOS gives you the degrees of freedom. You have not quoted it, but it won't be large in this case. With 12 variables, you start with 66 DF's, and subtract 1 for each parameter that you estimate. I don't know how many that would be, but you say that you have several factors and correlations between various constructs.

I do not entirely agree with Rolando2. In SEM, you gain by having lots of variables, assuming that they are reliable indicators of the underlying constructs. So don't reduce the number of variables. For the same reason, I do not entirely agree with @probabilityislogic. In SEM, you are not trying to model 12 variables with 42 observations. You are trying to model the constructs through 12 indicators, strengthened by 42 replications. A very simple factor model - 1 factor with 12 indicators - possibly could be tested with 42 people.

The RMSEA and other goodness of fit measures will tend to improve as you near saturation of the model, so again, you run the risk of a misleading result.

That being said, I have seen small data sets reject a factor model. It probably means something that the fit appears to be good.

Note: You can also check the residuals of a SEM model. These are the differences between the estimate covariance matrix and the model covariance matrix. AMOS will give them to you if you request them. Examination of the residuals might indicate if they are evenly distributed, or if certain covariances are very badly fitted.