14

You are using a flawed metric for the performance of the test by focusing only on the power to detect a difference, but ignoring the possibility of falsely rejecting the null hypothesis. If you just want to optimize this metric the "best" statistical test for you to use would be one that just returns p = 0 for all input data. You should replicate ...


11

This is a concern about the power of the Welch Test as compared to the textbook t test. Let's set up little simulation experiment and see if there are any discrepancies. The things that we can modulate are: The difference in means The sample size The difference in variance between groups, and If we use welch's or the regular t test. I'm going to modulate ...


5

I would just explain what happened. You powered for N, and you got N*. It's not the first time this has happened (and won't be the last). Post hoc power would not be especially useful (as you have realized.)


4

Elaborating a bit on Jeremy's answer, let's think for a minute about what a power analysis is. The purpose is to determine how many participants one would need to "detect" an effect of a specific size. So in discussing the results of your experiment vis a vis the sample size you originally designed, and what the pandemic (unforseen circumstances) ...


2

Inspired by the answer from @DemetriPananos and created a similar plot for various ratios of $\frac{n_1}{n_2}$ and $\frac{\sigma_1}{\sigma_2}$. These plots have the actual false positive rate on the x-axis and false negative rate on the y-axis for the two tests. In terms of statistical power, the two tests are near identical. The code for the plot below is ...


1

margin of error will be approximately $1.96\sqrt{\frac{0.8 \times 0.2}{n}}$. If you want the margin of error to be $m$, then take $n=1.96^2\frac{0.8 \times 0.2}{m^2}$. Example: 5% margin of error, $n=246$. The Wikipedia article also has a section for finite population size correction, but your population is pretty large and you would not need to worry about ...


1

If your results with 8 rats had been statistically significant, what would you have done? Would you have done a power analysis and determined you didn't have enough power and then run 11 more rats in each group? I suspect not. If you do this you are capitalizing on chance, and increasing your type I error rate. Your p-value no longer means what you think it ...


Only top voted, non community-wiki answers of a minimum length are eligible