I'm working on a paper involving multiple comparisons that have very small sample sizes (and no chance of increasing them). When I calculate p-values, they usually are not less than the usual .05-level, causing me to not reject the null hypothesis. However, I have very low power due to the small samples, thus can easily be committing Type II errors. Also, the effect size measures (e.g., Cramer's V) that I feel are meaningful have 95% CIs that are so large as to be irrelevant. Does anyone have any idea what I can report? Can I say anything about my comparisons?
$\begingroup$
$\endgroup$
9
-
5$\begingroup$ "We failed to find evidence that these quantities are different at the $\alpha$ level." $\endgroup$– AlexisCommented Jun 9, 2020 at 21:30
-
1$\begingroup$ @Alexis I'd qualify that even further by describing the low power of the study - failing to find a difference can be good evidence that there is no difference if you have enough samples to have a high-powered study. But in a low-powered study, failing to find evidence of a difference tells you very little (even if there was a difference, you wouldn't be likely to see it). Basically, low-power comparisons are doomed from the start unless you have huge effect sizes. $\endgroup$– Nuclear HoagieCommented Jun 9, 2020 at 21:40
-
2$\begingroup$ @Alexis. Depends on circumstances. OP may want to try again, this time with large enough $n$ to have adequate power, and may have info from current study that would help with that. Also, it may not burnish one's reputation to publish results of an obviously poorly planned study. (If I have the misfortune to make a huge noisy belch at a formal dinner party, I probably won't rush to share that on social media. Not quite an analogy, but I'd think twice about trying to publish nonsignificant results.) $\endgroup$– BruceETCommented Jun 9, 2020 at 23:48
-
2$\begingroup$ Thanx for all of your input. Unfortunately,the data I'm analyzing is not an experiment, but a study of a number of skeletons from 4 archaeological sites, hence the small sample sizes and low power. Thus, I had no control over how many individuals were recovered/recoverable. That said, can anyone think of anything I can report on these data? I hate not to publish as these skeletons are from a time period where little is known about the biology of the people, but I don't want to say the people were biologically similar from the 4 sites when I cannot support it with statistics. $\endgroup$– stevebyers2000Commented Jun 10, 2020 at 0:22
-
1$\begingroup$ With such small numbers, simply describe each individual's measures and qualities. $\endgroup$– AlexisCommented Jun 10, 2020 at 16:31
|
Show 4 more comments