Have the authors appropriately applied repeated measures one way ANOVA and ROC analysis in this test of therapist adherence?

Question

I'm reading a piece of research about the type of psychotherapy I practice. My intuition is that the use of statistics in the paper is flawed, but I am ideologically motivated to find flaws in this particular piece of research and my knowledge of statistics is fairly basic. For these reasons, I won't link to the paper.

All the authors of the paper are psychotherapists trained in the same type of therapy. They created a questionnaire to be used by an observer trained in that therapy to rate the degree to which a therapist worked within that therapy in a given session. (i.e. It is a test of adherence.) The authors distributed this questionnaire to about 200 participants (all trained in the therapy of interest) along with videos of 4 therapy sessions; 2 by therapists trained in the therapy of interest and 2 by therapists trained in another therapy. Participants scored the first two videos more highly (each approx. mean = 16.5, sd = 0.2) than the second two videos (each approx. mean = 4.5, sd = 0.25), indicating that use of the questionnaire was clearly distinguishing between therapy delivered by people trained in the therapy of interest and therapy delivered by people trained in the other therapy.

The authors presented the mean and sd values above as part of a "one way repeated measures ANOVA" test, which they said demonstrated statistical significance.

They then performed a ROC analysis to determine the threshold score that divided the therapy of interest from other types of therapy and got a value of 11.

I have two concerns:

1: Statology says that, "A repeated measures one-way ANOVA is used to determine whether or not there is a statistically significant difference between the means of three or more groups in which the same subjects show up in each group." I can see how there could be 4 groups here (1 group of scores for each of the 4 videos) but, taking Statology's wording literally, that would mean that the observer-raters are the subjects rather than the videoed sessions that they are rating. If the purpose of the research is to find a way to distinguish between the therapy of interest and other types of therapy then surely what we actually have is two groups (2 samples of the therapy of interest and 2 from another therapy.) If we only have 4 samples then I assume that tests of statistical significance are not yet appropriate. I am open to being told that I'm thinking about this incorrectly and would welcome any comments.

2: Wikipedia says that ROC analysis was developed in order to determine what strength of signal should be considered an object worth displaying on radar. I assume that task is necessarily binary in order to limit the amount of information that a radar operator has to parse. I don't see the applicability of a threshold to a scale to determine whether someone is adhering to a particular kind of therapy (especially if we only have 4 samples and we've only rated 2 types of therapy.) Again, I would welcome any comments, including any corrections of my perspective.

I don't know enough to determine whether I've posted enough information regarding the study. Please say if I should give more details.

Answer 1

In a repeated measures one-way ANOVA, the statistical model can be expressed as:

$$ Y_{ij} = \mu + \alpha_j + s_i + \varepsilon_{ij} $$

where:

• $Y_{ij}$ is the rating from subject $i$ (observer) in condition $j$ (video),
• $\mu$ is the overall mean,
• $\alpha_j$ is the fixed effect of the $j$-th condition,
• $s_i$ is the random effect for the $i$-th subject,
• $\varepsilon_{ij}$ is the residual error.

In this study, the 200 observers are the “subjects” ($i=1,\dots,200$), and the four videos represent the repeated conditions ($j=1,\dots,4$). The key assumption is that each subject provides a rating for each condition, so the ratings are correlated within subjects. The repeated measures ANOVA tests the null hypothesis $H_0: \alpha_1 = \alpha_2 = \alpha_3 = \alpha_4$. This is appropriate even with only four conditions, as long as the number of subjects is sufficient. With 200 subjects, the sample size is effectively 800 data points (200 per video). Thus, the test is statistically sound: it leverages within-subject comparisons to detect differences in means across conditions.

Note also that a ROC analysis takes a continuous measure $X$ and determines how well it discriminates between two known classes (here, “therapy of interest” vs. “other therapy”). For a given threshold $t$, we define classification as:

$$ \text{Classify as therapy of interest if } X > t; \text{ otherwise, classify as other therapy.} $$

By varying $t$, we obtain pairs $(\text{FPR}(t), \text{TPR}(t))$ where:

• $\text{TPR}(t) = P(X>t \mid \text{therapy of interest})$

• $\text{FPR}(t) = P(X>t \mid \text{other therapy})$

The ROC curve is the plot of $\text{TPR}(t)$ vs. $\text{FPR}(t)$ for all thresholds $t$. Although only four videos were shown (two from each category), each video was rated by 200 observers, yielding distributions of ratings from each category. Each rating is a draw from one of two distributions. The ROC analysis is therefore based on hundreds of data points per class, making it a valid statistical approach to estimate a threshold that discriminates between the two classes.

The repeated measures ANOVA is appropriate because the primary sampling units (subjects) each provide ratings under four conditions, allowing a within-subject comparison of means. The ROC analysis is appropriate because we have two distinct categories, a continuous measure, and a sufficiently large set of ratings to produce reliable estimates of sensitivity and specificity for various thresholds.