Whether to exclude a case from two analyses when the case is only an outlier in one?

Question

I have a question regarding the elimination of participant data for two statistical analyses on the same sample.

Here is a bit of background information:

I collected data from one sample (n = 181 college students) and administered two questionnaires to them (one on spirituality and one on negative emotional states). I performed an exploratory factor analysis on the data from the spirituality questionnaire, and I want to compare the factors derived from the EFA to the negative emotional states using a hierarchical multiple linear regression.

My main question is the following:

When I analyzed the data with a hierarchical multiple linear regression, SPSS identified (via case diagnostics) participants whose data could be eliminated due to outlier scores. If I exclude these cases from the analysis, then do I have to remove them from the results of the EFA as well? (Both of these analyses would be featured in the results section of the same experiment.)

Answer 1

You should start by asking yourself why the outliers "lie out"? If the values contain errors - like they were badly coded, or the respondent clearly didn't understand the process - then I would remove them from both parts of the analysis.

But if you believe that those data are sound, then what does it mean to say that there are outliers? Presumably, it means that the results for these individuals do not fit the model. And that is interesting to know. But you would not know that there were outliers if you had excluded them from both parts of your analysis.

It would be interesting to see what happens if you DO remove those outliers from the first part of the analysis. Do you get basically the same factor structure? Do you get a better fitting model for the second part? In other words, are the outliers driving the results?

At the end of the day, this is not so much a statistical question as a reporting question. How do you write up your study? To me, I think you need to write up the full data set, pointing out the outliers. Then, if you have a much better model, sans outliers, you could explain that, say, in 95% of the cases, your model explains a lot - but that a small percentage of individuals, spirituality and NES interact very differently from the norm.

Answer 2

If you are interested in the interaction between the results from the two questionnaires that data point may well skew results. If the analyses are separate then it may be that only that half of the data is an outlier - they may have understood half the process?