I've read the following paper about recall rates for information being higher when said information is supplemented with pictographs: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2656019/ and am a bit suspicious about some of the results, because I've never seen a significance test with $N=13$ (!) being signifcant with $p<0.001$ when it comes to percentage point differences.
So I'm referring to figure 2 of this paper, looking at the left part of the plot showing that the group with text only stimuli had a recall rate of 44.28% and the enhanced stimuli group a recall rate of 53.51%. And the authors claim that a
Mixed factor linear regression analysis found statistically significant effects $(P < 0.001)$ of the version (text vs. pictograph) on the recall rate.
And I really can't believe the results. Problem is that the authors aren't very explicate about their approach and the exact study setup (e.g. if the overall $N=13$, what is the $N$ for the text only and what is the $N$ for the enhanced stimuli group?). So it's difficult to make an assessment here. The only thing I could imagine is that each respondent got multiple different stimuli, so that the overall $N$ is higher than $13$ (e.g. if each respondent saw, let's say $20$ stimuli, the overall $N$ could have been $260$).
Also, I'm really not an expert in mixed effects models and maybe this all makes sense and results are correct, because
The effects of the version, cases, and the time lapse on the recall rates were tested with a linear mixed effect model where the instruction version, case, and the time when the recall rate was tested were set as fixed effect variables. Each respondent was set as a random effect variable and the recall rates was the response variable. This analysis was done using the proc mixed procedure with Statistical Analysis System (SAS) v9.1.
So maybe treating respondents as random effects might indeed lead to such low p-values?
In any case, it's hard for me to judge about the correctness of the results, so I'm hoping someone more familiar with mixed effects models could help me here?
Thanks.