# Mixed model effects - can the results in this paper be correct?

I've read the following paper about recall rates for information being higher when said information is supplemented with pictographs: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2656019/ and am a bit suspicious about some of the results, because I've never seen a significance test with $$N=13$$ (!) being signifcant with $$p<0.001$$ when it comes to percentage point differences.

So I'm referring to figure 2 of this paper, looking at the left part of the plot showing that the group with text only stimuli had a recall rate of 44.28% and the enhanced stimuli group a recall rate of 53.51%. And the authors claim that a

Mixed factor linear regression analysis found statistically significant effects $$(P < 0.001)$$ of the version (text vs. pictograph) on the recall rate.

And I really can't believe the results. Problem is that the authors aren't very explicate about their approach and the exact study setup (e.g. if the overall $$N=13$$, what is the $$N$$ for the text only and what is the $$N$$ for the enhanced stimuli group?). So it's difficult to make an assessment here. The only thing I could imagine is that each respondent got multiple different stimuli, so that the overall $$N$$ is higher than $$13$$ (e.g. if each respondent saw, let's say $$20$$ stimuli, the overall $$N$$ could have been $$260$$).

Also, I'm really not an expert in mixed effects models and maybe this all makes sense and results are correct, because

The effects of the version, cases, and the time lapse on the recall rates were tested with a linear mixed effect model where the instruction version, case, and the time when the recall rate was tested were set as fixed effect variables. Each respondent was set as a random effect variable and the recall rates was the response variable. This analysis was done using the proc mixed procedure with Statistical Analysis System (SAS) v9.1.

So maybe treating respondents as random effects might indeed lead to such low p-values?

In any case, it's hard for me to judge about the correctness of the results, so I'm hoping someone more familiar with mixed effects models could help me here?

Thanks.

• Number of people was 13. Number of instructions was 38. Number of rows in the dataset was 13 * 38 = 494. Effective sample size is determined by the intra-class correlations. Commented Sep 16, 2020 at 0:09
• (IIUC - I only skimmed the paper.) Commented Sep 16, 2020 at 0:09
• Thanks for looking into it! That's what I suspected as well, although it wasn't very clear in the paper, i.e. the 38 instructions sounded like that's the pool of instructions they created and that respondents might have been randomly assigned to one or multiple of these instructions. Reading further down below the authors get a bit more explicit with group A (15 instructions) and group B (19 instructions) and one could interpret this in a way that each respondent then has seen a total of 24 instructions. Although, again, the authors aren't explicate about it. Commented Sep 16, 2020 at 6:56

It appears that this study has repeated measures within patients, so $$N=13$$ refers to the number of patients, which is fine.