# Repeated Measures ANOVA post hoc test (bayesian)

I am trying to understand the procedure of carrying out a Bayesian Repeated Measures ANOVA. In a conventional repeated measures ANOVA, I calculate the effect of a certain parameter (e.g., study condition) on a measured variable (e.g., test score). Then, I follow up my test with a post hoc test if the p value is lower than a certain value. This post hoc test will tell me the effect of each individual study condition. Therefore, I do not only know that the study condition had an effect on the measured results - I am also able to infer which condition had the most effect.

For Bayesian Repeated Measures ANOVA, it is not clear to me how to follow up the initial test. I can do a 'model comparison', which gives me a Bayesian Factor to indicate the effect of that specific model. Lets say study condition is found to have a high BF - now I am of course interested which study condition performed better / worse than the others. How do I approach this question?

Additional information about the study: This concerns a within-subjects, full repeated measures design. The study consisted of a total of three conditions - with participants completing the different conditions in different orders (i.e., balanced latin square).

(I used JASP software for my analysis, whereas the regular repeated measures ANOVA offers the possibility for post hoc testing, this is not available in the Bayesian repeated measures ANOVA.)

• It might help to be a little more specific about your problem. For instance, do you have a fully repeated-measures design, and if so, how many conditions, and are the conditions purely categorical (i.e., there is no implied order)? Commented Nov 12, 2016 at 12:49
• @user1637894 Thank you for your comment - I have updated my question with additional information regarding the study. Commented Nov 12, 2016 at 13:46
• I'm in the process of formulating an answer to your question, but in order to provide a complete answer I need a few more details about your study. What form do the data take? One single, normally distributed value per subject per condition? Or do you have more than one observation per subject or condition, such as trials in an experiment? Commented Nov 12, 2016 at 18:12

First, why do you want to perform a Bayesian analysis? If you want to know which of your conditions is "significantly different" from the others, the repeated-measures ANOVA with planned contrasts gives you the answer. The strength of the Bayesian approach is not in this kind of null-hypothesis significance testing (NHST), but rather in parameter estimation. The basic idea is that under the Bayesian approach, you are never boiling down your model to binary decisions about accepting/rejecting models, nor assuming "point estimates" for your parameters, allowing you to evaluate the performance of your model in more detail. For more information, see Kruschke's excellent book or this paper, in particular the section entitled "Bayesian estimation generally". Previous questions on Cross Validated are well worth a read (here and here).

Now onto your specific problem. Let $n$ represent the number of subjects, $m$ represent the number of conditions (in this case, $m=3$), and $Y$ represent a stacked vector of your data, organized like this:

$$Y = \begin{pmatrix} y_{1} \\ y_{2} \\ \vdots \\ y_{n\cdot{}m} \end{pmatrix},$$

where the $y_1$ to $y_m$ are the values from the first subject, $y_{m+1}$ to $y_{2m}$ are the values from the second subject, and so on. We assume that the data are normally distributed,

$$Y\sim\mathcal{N}\left(\vec{\mu},\sigma^2\right),$$

where $\vec{\mu}$ is a stacked vector of means and $\sigma^2$ is the variance. The mean vector is given by

$$\vec{\mu}=\textbf{X}\vec{\beta},$$

where $\textbf{X}$ is a design matrix and $\vec{\beta}$ is a row vector of regression coefficients. The design matrix is a bit complicated due to the repeated-measures design. Each row represents a data point in $Y$. The first $n$ columns in the matrix represent subject-specific "intercepts"; for example, in the first column, the first three values are 1 and all the rest are 0. The remaining columns represent conditions.The first condition is absorbed by the intercept, so the $(n+1)$th column has a 1 for every data point belonging to condition 2, the $(n+2)$th column has a 1 for every data point belonging to condition 3, and so on.

The row vector $\vec{\beta}$ looks like

$$\vec{\beta} = \begin{pmatrix} \beta_1 & \beta_2 & \cdots & \beta_{n\cdot{}m-1} \end{pmatrix},$$

where each entry is a normally distributed random variable with a suitable prior. Finally, we need to assign a prior to $\sigma$, such as a half-Cauchy.

And that's it! You can construct this model in BUGS, JAGS, Stan, or PyMC, and sample the joint posterior using MCMC. To determine whether there was a difference between, for example, condition 1 and condition2, look at the marginal posterior distribution of $\beta_{n+1}$. This represents the group-level average difference between these conditions. If the 95% credible interval of this distribution excludes 0, you can conclude that there is probably a meaningful difference between condition 1 and condition 2.

PS: I really, really hate the term "Bayesian ANOVA". I know it crops up all the time and there is nothing wrong with it technically, but I find that for the uninitiated, it implies a too much equivalence between the frequentist and Bayesian approaches.

• Please, do not "frequentise" the Bayesian approach! Bayesian testing cannot be performed by credible intervals but only through Bayes Factors. Commented Nov 12, 2016 at 21:23
• Not sure I follow your point. What's wrong with credible intervals? Commented Nov 12, 2016 at 21:54
• I would consider, strongly, using brms package in r. Commented Nov 12, 2016 at 22:04
• @user1637894 essentially Bayesian testing via credible intervals lacks a decision-theoretic foundation; there are plenty of books on Bayesian analysis which explain how Bayesian testing is to be done (via BF). See, for instance, The Bayesian Choice, by Robert; Bayesian Theory by Bernardo&Smith, An Introduction to Bayesian Analysis by Gosh, Dalampady and Samanta, etc. Commented Nov 12, 2016 at 22:15
• Right. But plenty of statisticians do advocate the use of CIs for parameter estimation. Seems to me that parameter estimation/CIs and hypothesis testing/BFs reflect alternative approaches. Not sure it was worth a down vote. I will happily amend my comment to reflect this. Commented Nov 13, 2016 at 10:57