Timeline for Baffling results in the simulation of a power analysis of a sequential, online A/B experiment
Current License: CC BY-SA 4.0
9 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Mar 10, 2022 at 5:44 | comment | added | RAFisherman | 1) Good point, thanks! 2) Thanks - that last point reflected my true misunderstanding. This caused me to re-run the above simulation with different alpha levels. When I do that, the intervals don't "stop overlapping" at 80% power, so my theory was just a coincidence. Instead, I should probably simulate the confidence interval of the difference. | |
| Mar 10, 2022 at 5:43 | vote | accept | RAFisherman | ||
| Mar 7, 2022 at 7:37 | comment | added | num_39 | 2) " If comparing non-overlapping quantiles reflects the power of an exact test, why do they intersect perfectly where I'd achieve 80% power for an approximate test?" Non-overlapping quantiles are not an exact test of power. For the relation between confidence intervals and tests of significance, see towardsdatascience.com/… | |
| Mar 7, 2022 at 7:20 | comment | added | num_39 | 1) When you take thousands of replications for g1, your empirical quantiles will essentially match the theoretical quantiles for the fixed proportion of 0.7. So it's as if you're comparing g0 to a fixed proportion, which is a binomial test. | |
| Mar 7, 2022 at 6:57 | history | edited | RAFisherman | CC BY-SA 4.0 |
added 2138 characters in body; edited tags
|
| Mar 6, 2022 at 22:54 | history | edited | RAFisherman | CC BY-SA 4.0 |
Reversed power and beta typo
|
| Mar 6, 2022 at 12:27 | answer | added | num_39 | timeline score: 1 | |
| S Mar 6, 2022 at 7:18 | review | First questions | |||
| Mar 6, 2022 at 14:37 | |||||
| S Mar 6, 2022 at 7:18 | history | asked | RAFisherman | CC BY-SA 4.0 |