# Follow up: In a mixed within-between ANOVA plot estimated SEs or actual SEs?

I am currently finishing a paper and stumbled upon this question from yesterday which led me to pose the same question to myself. Is it better to provide my graph with the actual standard error from the data or the one estimated from my ANOVA?
As the question from yesterday was rather unspecific and mine is pretty specific I thought it would be appropriate to pose this follow-up question.

Details:
I have run an experiment in some cognitive psychology domain (conditional reasoning) comparing two groups (inductive and deductive instructions, i.e., a between-subjects manipulation) with two within-subjects manipulations (problem type and content of the problem, each with two factor levels).

The results look like this (left panel with SE-estimates from the ANOVA Output, right panel with SEs estimated from the data):
Note that the different lines represent the two different groups (i.e., the between-subjects manipulation) and the within-subjects manipulations are plotted on the x-axis (i.e., the 2x2 factor levels).

In the text I provide the respective results of the ANOVA and even planned comparisons for the critical cross-over interaction in the middle. The SEs are there to give the reader some hint about the variability of the data. I prefer SEs over standard deviations and confidence intervals as it is not common to plot SDs and there are severe problems when comparing within- and between-subjects CIs (as the same surely applys for SEs, it is not so common to falsely infer significant differences from them).

To repeat my question: Is it better to plot the SEs estimated from the ANOVA or should I plot the SEs estimated from the raw data?

Update:
I think I should be a little bit clearer in what the estimated SEs are. The ANOVA Output in SPSS gives me estimated marginal means with corresponding SEs and CIs. This is what is plotted in the left graph. As far as I understand this, they should be the SDs of the residuals. But, when saving the residuals their SDs are not somehow near the estimated SEs. So a secondary (potentially SPSS specific) question would be:
What are these SEs?

UPDATE 2: I finally managed to write a R-function that should be able to make a plot as I finally liked it (see my accepted answer) on its own. If anybody has time, I would really appreciate if you could have a look at it. Here it is.

-
Can you clarify the predicted variable, "mean endorsement"?. Is this a 0-100 scale that participants used for response, or is this a measure of the proportion of trials on which participants said "yes, I endorse" (vs. "no, I do not endorse"). If the latter, then it is inappropriate to analyse this data as proportions. Instead, you should be analyzing the raw, trial-by-trial data using a mixed effects model with a binomial link function. –  Mike Lawrence Aug 24 '10 at 11:36
Sorry, for omitting this: it is a 0-100 response scale. –  Henrik Aug 24 '10 at 11:39
Do you have many 0's or 100's? If not, I'd consider dividing by 100 and performing a logit transform to take into account the restriction of range at the extremes. This is essentially what is achieved by the binomial link function when you have binary data, but is useful if you only have proportion-like data as you seem to have here. However, you can't logit transform 1 or 0, so you'd have to toss any responses of 100 or 0. –  Mike Lawrence Aug 24 '10 at 12:16
Oops, just realized that my first comment was not 100% correct. Each plotted mean represents the mean of two responses on a 0-100 scale. In this data there are a lot values very near to 100, and some directly on 100, but actually very little at 0 and around 0. You have some literature for justifing your recommendation? –  Henrik Aug 24 '10 at 12:28
Other data visualization people might claim that bar graphs are a crime against humanity :Op –  Mike Lawrence Aug 24 '10 at 17:52
show 1 more comment

You will not find a single reasonable error bar for inferential purposes with this type of experimental design. This is an old problem with no clear solution.

It seems impossible to have the estimate SE's you have here. There are two main kinds of error in such a design, the between and within S error. They are usually very different from one another and not comparable. There just really is no good single error bar to represent your data.

One might argue that the raw SEs or SDs from the data are most important in a descriptive rather than inferential sense. They either tell about the quality of the central tendency estimate (SE) or the variability of the data (SD). However, even then it's somewhat disingenuous because the thing you're testing and measuring within S is not that raw value but rather the effect of the within S variable. Therefore, reporting variability of the raw values is either meaningless or misleading with respect to within S effects.

I have typically endorsed no error bars on such graphs and adjacent effects graphs indicating the variability of the effects. One might have CI's on that graph that are perfectly reasonable. See Masson & Loftus (2003) for examples of the effects graphs. Simply eliminate their ((pretty much completely useless) error bars around the mean values they show and just use the effect error bars.

For your study I'd first replot the data as the 2 x 2 x 2 design it is (2-panel 2x2) and then plot immediately adjacent a graph with confidence intervals of the validity, plausibility, instruction, and interaction effects. Put SDs and SEs for the instruction groups in a table or in the text.

(waiting for expected mixed effects analysis response ;) )

UPDATE: OK, after editing it's clear the only thing you want is an SE to be used to show the quality of the estimate of the value. In that case use your model values. Both values are based on a model and there is no 'true' value in your sample. Use the ones from the model you applied to your data. BUT, make sure you warn readers in the figure caption that these SEs have no inferential value whatsoever for your within S effects or interactions.

UPDATE2: Looking back at the data you did present... that looks suspiciously like percentages which shouldn't have been analyzed with ANOVA in the first place. Whether it is or isn't, it's a variable that maxes at 100 and has reduced variances at the extremes so it still shouldn't be analyzed with ANOVA. I do very much like your rm.plot plots. I'd still be tempted to do separate plots of the between conditions, showing the raw data, and within conditions showing the data with between S variability removed.

-
I have good (non-statistical) reasons to plot the graph as it is: You directly see the answer to the research question. Furthermore, I am not looking for a error bars for inferential purposes as I know about the within-between problems. But, thanks to pinpointing me back to Mason & Loftus, I must have forgotten that they had a mixed example. I have to think on whether or not it serves my purpose. –  Henrik Aug 24 '10 at 12:16

This looks like a very nice experiment, so congratulations!

I agree with John Christie, it is a mixed model, but provided it can be properely specified in an ANOVA design (& is balanced) I don't see why it can't be so formulated. Two factor within and 1-factor between subjects, but the between subjects factor (inductive/deductive) clearly interacts (modifies) the within-subjects effects. I assume the plotted means are from the ANOVA model (LHS) and so the model is correctly specified. Well done - this is non-trivial!

Some points: 1) The "estimated" vs "actual" "error" is a false dichotomy. Both assume an underlying model and make estimates on that basis. If the model is reasonable, I would argue it is better to use the model-based estimates (they are based on the pooling of larger samples). But as James mentions, the errors differ depending on the comparison you are making, so no simple representation is possible.

2) I would prefer to see box-plots or individual data points plotted (if there are not too many), perhaps with some sideways jitter, so points with the same value can be distinguished.

http://en.wikipedia.org/wiki/Box_plot

3) If you must plot an estimate of the error of the mean, never plot SDs - they are an estimate of the standard deviation of the sample and relate to population variablility, not a statistical comparison of means. It is generally preferable to plot 95% confidence intervals rather than SEs, but not in this case (see 1 and John's point)

4) The one issue with this data that concerns me is the assumption of uniform variance is probably violated as the "MP Valid and Plausible" data are clearly constrained by the 100% limit, especially for the deductive people. I'm tossing up in my own mind how important this issue is. Moving to a mixed-effects logit (binomial probability) is probably the ideal solution, but it's a hard ask. It might be best to let others answer.

-
I am not quite sure I understand your recommendation in 1. As actual SE [i.e., SD/sqrt(n)] and estimated SE are both model based, you recommend using the model-based. So which one? Or do you mean: go with the more complicated model (here: ANOVA) cause both models are reasonable. –  Henrik Aug 24 '10 at 12:07
agree with point 1 completely –  John Aug 24 '10 at 12:09
Hi Henrik, Simple example - compare two groups (x1, x2) assumed ND. Assumptions & models: 1) Independently sampled, different variance. SEs for x1, x2 estimated separately. This is implicitly the assumption in many graphical presentations. The estimated SEs differ. 2) Indep., same var. Usual ANOVA assumption. Estimate SEs using pooled RSS. Estimate is more robust IF assumptions correct. 3) Each x1 has an x2 pair. SEs estimated from x1-x2. To effectively plot them you need to plot the difference x1-x2. Once you mix 1) and 2) you have a real problem plotting meaningful SEs or CIs. –  Thylacoleo Aug 25 '10 at 0:42
Henrik, a comment on the plot. How many subjects do you have? I would strongly recommend plotting the data individually and use line segments to link individuals. (Line segments linking means is deceptive.) There is no need to plot SEs. The idea is to visually support your statistical analysis. Provided the plot does not become too cluttered, a reader should see (for example) that the clear majority of scores go up from MP-valid-implaus to AC-inval-plaus for the Inductive group & down for the Deductive group. See: jstor.org/stable/2685323?seq=1 Especially Figs 1 & 9 bottom panels. –  Thylacoleo Aug 25 '10 at 2:01

As a consequence of the inspiring answers and discussion to my question I constructed the following plots that do not rely on any model based parameters, but present the underlying data.

The reasons are that independent of whatever kind of standard-error I may choose, the standard-error is a model based parameter. So, why not present the underlying data and thereby transmit more information?

Furthermore, if choosing the SE from the ANOVA, two problems arise for my specific problems.
First (at least for me) it is somehow unclear what the SEs from SPSS ANOVA Output actually are (see also this discussion, in the comments). They are somehow related to the MSE but how exactly I don't know.
Second, they are only reasonable when the underlying assumptions are met. However, as the following plots show, the assumptions of homogeneity of variance is clearly violated.

The plots with boxplots:

The plots with all data points:

Note that the two groups are dislocated a little to the left or the right: deductive to the left, inductive to the right. The means are still plotted in black and the data or boxplots in the background in grey. The differences between the plots on the left and on the right are if the means are dislocated the same as the points or boxplots or if they are presented centrally.
Sorry for the nonoptimal quality of the graphs and the missing x-axis labels.

The question that remains is, which one of the above plots is the one to choose now. I have to think about it and ask the other author of our paper. But right now, I prefer the "points with means dislocated". And I still would be very interested in comments.

Update: After some programming I finally managed to write a R-function to automatically create a plot like points with means dislocated. Check it out (and send me comments)!

-
Excellent Henrik. I also prefer the "points with means dislocated". Linking subjects with line segments may look too cluttered. Pity. As to homogeneity of variance I am a little more sanguine. The variance problem may not be as bad as it looks in the raw data. For the most part I suspect you will be comparing contrasts - within group differences. Contrast variances will be more homogeneous that the variances of the raw data. If raw measures with different variances are compared (eg. Inductive vs Deductive in the MP-valiad & plausible group) a non-parametric test could be used as a back-up. –  Thylacoleo Aug 26 '10 at 12:58
I like the points with mean centrally. It has a truer representation of the lines. You could make the points smaller. –  John Aug 29 '10 at 19:18

Lately I've been using mixed effects analysis, and in attempting to develop an accompanying visual data analysis approach I've been using bootstrapping (see my description here), which yields confidence intervals that are not susceptible to the within-versus-between troubles of conventional CIs.

Also, I would avoid mapping multiple variables to the same visual aesthetic, as you have done in the graph above; you have 3 variables (MP/AC, valid/invalid, plausible/implausible) mapped to the x-axis, which makes it rather difficult to parse the design and patterns. I would suggest instead mapping, say, MP/AC to the x-axis, valid/invalid to facet columns, and plausible/implausible to facet rows. Check out ggplot2 in R to easily achieve this, eg:

library(ggplot2)
ggplot(
data = my_data
, mapping = aes(
y = mean_endorsement
, x = mp_ac
, linetype = deductive_inductive
, shape = deductive_inductive
)+
geom_point()+
geom_line()+
facet_grid(
plausible_implausible ~ valid_invalid
)

-
Mike, in the package languageR the pvals.fnc function does an MCMC to evaluate the hypotheses of the lmer model - however it does not handle designs with random slopes - that lead me to suspect that there was some reason doing MCMC with random slopes was in someway problematic, do you know definitively that there is no such problem? –  rpierce Aug 24 '10 at 13:32
I have to admit I still haven't figured out how MCMC works, which is one of the reasons I opted for bootstrapping instead. While bootstrapping should be possible with random slopes, as you intimated, it may be that pvals.fnc doesn't let you do CIs for models with random slopes because this is for some reason invalid, and further it may be that this invalidity extends to bootstrapping such models. I don't intuitively think there would be any problem with bootstrapping, but that may be a function of my limited expertise. –  Mike Lawrence Aug 24 '10 at 14:28