5
$\begingroup$

I've read the following paper about recall rates for information being higher when said information is supplemented with pictographs: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2656019/ and am a bit suspicious about some of the results, because I've never seen a significance test with $N=13$ (!) being signifcant with $p<0.001$ when it comes to percentage point differences.

So I'm referring to figure 2 of this paper, looking at the left part of the plot showing that the group with text only stimuli had a recall rate of 44.28% and the enhanced stimuli group a recall rate of 53.51%. And the authors claim that a

Mixed factor linear regression analysis found statistically significant effects $(P < 0.001)$ of the version (text vs. pictograph) on the recall rate.

And I really can't believe the results. Problem is that the authors aren't very explicate about their approach and the exact study setup (e.g. if the overall $N=13$, what is the $N$ for the text only and what is the $N$ for the enhanced stimuli group?). So it's difficult to make an assessment here. The only thing I could imagine is that each respondent got multiple different stimuli, so that the overall $N$ is higher than $13$ (e.g. if each respondent saw, let's say $20$ stimuli, the overall $N$ could have been $260$).

Also, I'm really not an expert in mixed effects models and maybe this all makes sense and results are correct, because

The effects of the version, cases, and the time lapse on the recall rates were tested with a linear mixed effect model where the instruction version, case, and the time when the recall rate was tested were set as fixed effect variables. Each respondent was set as a random effect variable and the recall rates was the response variable. This analysis was done using the proc mixed procedure with Statistical Analysis System (SAS) v9.1.

So maybe treating respondents as random effects might indeed lead to such low p-values?

In any case, it's hard for me to judge about the correctness of the results, so I'm hoping someone more familiar with mixed effects models could help me here?

Thanks.

Improve this question
$\endgroup$
3
  • 3
    $\begingroup$ Number of people was 13. Number of instructions was 38. Number of rows in the dataset was 13 * 38 = 494. Effective sample size is determined by the intra-class correlations. $\endgroup$
    – Jeremy Miles
    Commented Sep 16, 2020 at 0:09
  • $\begingroup$ (IIUC - I only skimmed the paper.) $\endgroup$
    – Jeremy Miles
    Commented Sep 16, 2020 at 0:09
  • $\begingroup$ Thanks for looking into it! That's what I suspected as well, although it wasn't very clear in the paper, i.e. the 38 instructions sounded like that's the pool of instructions they created and that respondents might have been randomly assigned to one or multiple of these instructions. Reading further down below the authors get a bit more explicit with group A (15 instructions) and group B (19 instructions) and one could interpret this in a way that each respondent then has seen a total of 24 instructions. Although, again, the authors aren't explicate about it. $\endgroup$
    – deschen
    Commented Sep 16, 2020 at 6:56

1 Answer 1

Reset to default
6
$\begingroup$

It appears that this study has repeated measures within patients, so $N=13$ refers to the number of patients, which is fine.

Patients are then observed 38 times each ("instructions" using their terminology).

It seems perfectly reasonable to find a low p value in such a dataset, however please try not to be too concerned with p values. Effect sizes are just as, if not more, important. In this study I would be concerned about how they decided to choose 13 patients. There is no mention of a prior power analysis which is very important. The study could be sereverly under powered and I see the authors do mention the small sample size in the discussion.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks for your answer. So I guess we all agree that the „rows“ in their data set are not 13, but sth. considerbaly larger. If we are talking about a N of 400+, p values could indeed be that low. I also agree that p values are not the only thing we need to look at, they just caught my attention here given their really low number of respondents. Letting the p value aside there are certainly other things to worry about here, true. $\endgroup$
    – deschen
    Commented Sep 16, 2020 at 13:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.