I am currently running some mixed ANOVAs for a 2x2 design (two groups, with pre- and post- test). Some of my variable are fine, but some of them violate assumptions for parametric testing, e.g. non-normal, skewed, kurtosis. Furthermore, my sample is not really large enough to be assuming I can ignore violations due to the central-limit theorem: I have 24 in each group, so 48 at pre- and 48 at post-.

As a result, I am running my mixed using the WRS2 package in r, specifically with the bwtrim() function, which works on trimmed means (I am using a trimming of 10% each side) and with the sppba(), sppbb(), and sppbi() functions, which work by boot-strapping the results. However, when I do this, I do not get equivalent results with the two methods. For example, when I run bwtrim on my variable ABmelt, I get:

Results of the bwtrim function, showing pre-post significant main effect, p=0.03 so we can see that there is a significant pre-post main effect change, but no main effect of group, and no interaction term. However, when I run the sppbb() function, I do NOT get a significant main effect of time:

Call: sppbb(formula = OUTCOME_VARIABLE ~ GROUPING_VARIABLE * PrePost, id = ID, data = my_data)

Test statistics: Estimate PRE-POST -0.2057

Test whether the corresponding population parameters are the same: p-value: 0.134

In general, my results using the trimmed means are much more similar to my results running "normal" (non-robust) mixed ANOVA. So my question is this: which robust method should I be using? Is it a trade-off with robust-ness and power, or do we base or decision based on properties of the distributions?

In the published WRS2 papers (e.g. see here: https://dornsife.usc.edu/assets/sites/239/docs/WRS2.pdf) they seem to be using the bootstrapping primarily to look at more complex mixed-ANOVAs where there are more groups (3x4 design), but that doesn't really seem to be a reason to use one over the other.

For harmony across my analyses, I was wanting to use a robust method across all of my variables (I have around 10)- from those violating all assumptions (highly skewed, ceiling effect) to those that aren't really that problematic. Of course, if this is making me lose too much power, then I run into the risk of type II (instead of type I) error.

Any thoughts and suggestions appreciated! Thank you :)

  • 1
    $\begingroup$ Bootstrapping in and of itself is not related to robustness, however if you are estimating a statistic using a mean inside bootstrapping than this is not robust, regardless of bootstrapping, so it is not that surprising that the results differ. $\endgroup$ Jul 18 at 11:27
  • $\begingroup$ @user2974951- do you recommend just using trimmed means then? $\endgroup$ Jul 18 at 11:35

1 Answer 1


Trimmed means are difficult to interpret in my opinion, and may not always be statistically efficient. I think that a better choice is the use of semi-parametric regression models such as the proportional odds ordinal logistic regression model, which generalizes the Wilcoxon and related rank tests. Semiparametric models use only the ordering of $Y$, not the absolute values, and are robust to extreme values. They also have the great advantage of transformation invariance, i.e., the regression coefficients and statistical test statistics don't change if you replace $Y$ with $\log(Y), \sqrt(Y)$, etc. For resources see https://www.fharrell.com/post/rpo .

After fitting an ordinal model, especially when using the R rms package's orm function, you can obtain predicted means, quantiles, and exceedance probabilities and their differences over values of $X$.

  • 1
    $\begingroup$ Thank you very much @frankharrel! I will have a look at fitting an ordinal model and hopefully will be able to get it work. I will post any findings here! $\endgroup$ Jul 18 at 11:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.