0
$\begingroup$

I have performed a univariate Cox proportional hazards regression for an admittedly miniscule group of 25 samples, and consequently received a certain p-value (significant) and an associated confidence interval for the hazard ratio. The goal is not to create an applicable predictor or classifier, but rather to simply find out whether the variable of interest could be associated with the outcome. Now I need to decide whether or not this sample size is sufficient to trust the test, because a reviewer has asked about it. How do I convince either myself that the test is unreliable at this sample size, or, alternatively, the reviewer that the sample size is sufficient to trust the p-value and the confidence interval? I have looked into post hoc power analysis, but find that all methods seem to be designed for two group comparisons. There are also plenty of opinions that post hoc tests are meaningless to begin with.

I am not looking for an opinion about whether or not the size is sufficient, but rather for methods or references that would allow one to determine a minimum sample size required to trust the analysis. There is a significant riskt that the question is misguided in some way, due to a lack of knownledge, but it would good to know why it is misguided in that case.

$\endgroup$
1
$\begingroup$

There are also plenty of opinions that post hoc tests are meaningless to begin with.

Not an opinion, it is demonstrable fact. A post hoc power analysis doesn't tell you anything the p-value doesn't tell you. It is considered a huge faux pas to include on of these in your analysis, and your reviewer should be made aware of that in a polite and respectful way. See my previous answer here.

There must be some reason why you proceeded with the analysis. I think at this stage, being honest with the reviewer is all you can do. Trying to back pedal and justify the same size with existing studies might help, but I think it would be very dishonest.

$\endgroup$
  • $\begingroup$ Well, if your p-value is significant, the data provide sufficient evidence that the predictor in question has an effect on the hazard you are interested in. Have a look at the estimated hazard - how far is it from 1? If it is quite far from 1, this effect is substantial in practical terms. If it is quite close to 1, this effect may be too small to be of any consequence in practice. Next, look at the width of the confidence interval - is it fairly narrow or quite wide? If it is quite wide, your effect estimate is imprecise - including more subjects in a future study would improve precision. $\endgroup$ – Isabella Ghement Jan 15 at 0:28
  • $\begingroup$ Of course, this is a one-off study so you shouldn't necessarily jump up and down with joy - perhaps replication of the study under similar conditions might be necessary. $\endgroup$ – Isabella Ghement Jan 15 at 0:30
  • $\begingroup$ How much you trust the study boils down to how many subjects you include in the study - the more subjects, the more you trust the results. But you are in a happy place because you do have evidence of a statistically significant effect - you just need to determine if the magnitude of that effect is large enough to matter in practice. You also need to comment on the precision of your estimated effect by noting how wide the confidence interval is. $\endgroup$ – Isabella Ghement Jan 15 at 0:32
  • $\begingroup$ Since you didn't conduct an a priori sample size calculation, you lucked out to find a significant effect. Usually, people would resort to an ad hoc power calculation in situations were they couldn't find a significant effect (despite the fact that this calculation is highly controversial). $\endgroup$ – Isabella Ghement Jan 15 at 0:34
  • 1
    $\begingroup$ @IsabellaGhement This is along the line of my original reasoning. Given the results, the test was obviously powerful enough to detect the difference. The only problem is that the small sample size prevents in the generalization of the results. But the statistics of the test itself will probably not fundamentally break down at this sample size. In this case we are honest about this fact when we present the results, and state that generalizeability is low and that this association needs to be validated in a future larger cohort. But counter arguments against this reasoning are welcome. $\endgroup$ – zbox Jan 15 at 11:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.