Follow up: In a mixed within-between ANOVA plot estimated SEs or actual SEs?

Question

I am currently finishing a paper and stumbled upon this question from yesterday which led me to pose the same question to myself. Is it better to provide my graph with the actual standard error from the data or the one estimated from my ANOVA?
As the question from yesterday was rather unspecific and mine is pretty specific I thought it would be appropriate to pose this follow-up question.

Details:
I have run an experiment in some cognitive psychology domain (conditional reasoning) comparing two groups (inductive and deductive instructions, i.e., a between-subjects manipulation) with two within-subjects manipulations (problem type and content of the problem, each with two factor levels).

The results look like this (left panel with SE-estimates from the ANOVA Output, right panel with SEs estimated from the data):
Note that the different lines represent the two different groups (i.e., the between-subjects manipulation) and the within-subjects manipulations are plotted on the x-axis (i.e., the 2x2 factor levels).

In the text I provide the respective results of the ANOVA and even planned comparisons for the critical cross-over interaction in the middle. The SEs are there to give the reader some hint about the variability of the data. I prefer SEs over standard deviations and confidence intervals as it is not common to plot SDs and there are severe problems when comparing within- and between-subjects CIs (as the same surely applys for SEs, it is not so common to falsely infer significant differences from them).

To repeat my question: Is it better to plot the SEs estimated from the ANOVA or should I plot the SEs estimated from the raw data?

Update:
I think I should be a little bit clearer in what the estimated SEs are. The ANOVA Output in SPSS gives me estimated marginal means with corresponding SEs and CIs. This is what is plotted in the left graph. As far as I understand this, they should be the SDs of the residuals. But, when saving the residuals their SDs are not somehow near the estimated SEs. So a secondary (potentially SPSS specific) question would be:
What are these SEs?

---

UPDATE 2: I finally managed to write a R-function that should be able to make a plot as I finally liked it (see my accepted answer) on its own. If anybody has time, I would really appreciate if you could have a look at it. Here it is.

Answer 1

As a consequence of the inspiring answers and discussion to my question I constructed the following plots that do not rely on any model based parameters, but present the underlying data.

The reasons are that independent of whatever kind of standard-error I may choose, the standard-error is a model based parameter. So, why not present the underlying data and thereby transmit more information?

Furthermore, if choosing the SE from the ANOVA, two problems arise for my specific problems.
First (at least for me) it is somehow unclear what the SEs from SPSS ANOVA Output actually are (see also this discussion, in the comments). They are somehow related to the MSE but how exactly I don't know.
Second, they are only reasonable when the underlying assumptions are met. However, as the following plots show, the assumptions of homogeneity of variance is clearly violated.

The plots with boxplots:

The plots with all data points:

Note that the two groups are dislocated a little to the left or the right: deductive to the left, inductive to the right. The means are still plotted in black and the data or boxplots in the background in grey. The differences between the plots on the left and on the right are if the means are dislocated the same as the points or boxplots or if they are presented centrally.
Sorry for the nonoptimal quality of the graphs and the missing x-axis labels.

The question that remains is, which one of the above plots is the one to choose now. I have to think about it and ask the other author of our paper. But right now, I prefer the "points with means dislocated". And I still would be very interested in comments.

---

Update: After some programming I finally managed to write a R-function to automatically create a plot like points with means dislocated. Check it out (and send me comments)!

Answer 2

You will not find a single reasonable error bar for inferential purposes with this type of experimental design. This is an old problem with no clear solution.

It seems impossible to have the estimate SE's you have here. There are two main kinds of error in such a design, the between and within S error. They are usually very different from one another and not comparable. There just really is no good single error bar to represent your data.

One might argue that the raw SEs or SDs from the data are most important in a descriptive rather than inferential sense. They either tell about the quality of the central tendency estimate (SE) or the variability of the data (SD). However, even then it's somewhat disingenuous because the thing you're testing and measuring within S is not that raw value but rather the effect of the within S variable. Therefore, reporting variability of the raw values is either meaningless or misleading with respect to within S effects.

I have typically endorsed no error bars on such graphs and adjacent effects graphs indicating the variability of the effects. One might have CI's on that graph that are perfectly reasonable. See Masson & Loftus (2003) for examples of the effects graphs. Simply eliminate their ((pretty much completely useless) error bars around the mean values they show and just use the effect error bars.

For your study I'd first replot the data as the 2 x 2 x 2 design it is (2-panel 2x2) and then plot immediately adjacent a graph with confidence intervals of the validity, plausibility, instruction, and interaction effects. Put SDs and SEs for the instruction groups in a table or in the text.

(waiting for expected mixed effects analysis response ;) )

UPDATE: OK, after editing it's clear the only thing you want is an SE to be used to show the quality of the estimate of the value. In that case use your model values. Both values are based on a model and there is no 'true' value in your sample. Use the ones from the model you applied to your data. BUT, make sure you warn readers in the figure caption that these SEs have no inferential value whatsoever for your within S effects or interactions.

UPDATE2: Looking back at the data you did present... that looks suspiciously like percentages which shouldn't have been analyzed with ANOVA in the first place. Whether it is or isn't, it's a variable that maxes at 100 and has reduced variances at the extremes so it still shouldn't be analyzed with ANOVA. I do very much like your rm.plot plots. I'd still be tempted to do separate plots of the between conditions, showing the raw data, and within conditions showing the data with between S variability removed.

Answer 3

This looks like a very nice experiment, so congratulations!

I agree with John Christie, it is a mixed model, but provided it can be properely specified in an ANOVA design (& is balanced) I don't see why it can't be so formulated. Two factor within and 1-factor between subjects, but the between subjects factor (inductive/deductive) clearly interacts (modifies) the within-subjects effects. I assume the plotted means are from the ANOVA model (LHS) and so the model is correctly specified. Well done - this is non-trivial!

Some points: 1) The "estimated" vs "actual" "error" is a false dichotomy. Both assume an underlying model and make estimates on that basis. If the model is reasonable, I would argue it is better to use the model-based estimates (they are based on the pooling of larger samples). But as James mentions, the errors differ depending on the comparison you are making, so no simple representation is possible.

2) I would prefer to see box-plots or individual data points plotted (if there are not too many), perhaps with some sideways jitter, so points with the same value can be distinguished.

http://en.wikipedia.org/wiki/Box_plot

3) If you must plot an estimate of the error of the mean, never plot SDs - they are an estimate of the standard deviation of the sample and relate to population variablility, not a statistical comparison of means. It is generally preferable to plot 95% confidence intervals rather than SEs, but not in this case (see 1 and John's point)

4) The one issue with this data that concerns me is the assumption of uniform variance is probably violated as the "MP Valid and Plausible" data are clearly constrained by the 100% limit, especially for the deductive people. I'm tossing up in my own mind how important this issue is. Moving to a mixed-effects logit (binomial probability) is probably the ideal solution, but it's a hard ask. It might be best to let others answer.

Answer 4

Lately I've been using mixed effects analysis, and in attempting to develop an accompanying visual data analysis approach I've been using bootstrapping (see my description here), which yields confidence intervals that are not susceptible to the within-versus-between troubles of conventional CIs.

Also, I would avoid mapping multiple variables to the same visual aesthetic, as you have done in the graph above; you have 3 variables (MP/AC, valid/invalid, plausible/implausible) mapped to the x-axis, which makes it rather difficult to parse the design and patterns. I would suggest instead mapping, say, MP/AC to the x-axis, valid/invalid to facet columns, and plausible/implausible to facet rows. Check out ggplot2 in R to easily achieve this, eg:

library(ggplot2)
ggplot(
    data = my_data
    , mapping = aes(
        y = mean_endorsement
        , x = mp_ac
        , linetype = deductive_inductive
        , shape = deductive_inductive
)+
geom_point()+
geom_line()+
facet_grid(
    plausible_implausible ~ valid_invalid
)