Timeline for Target population for power analysis of ML model A/B test
Current License: CC BY-SA 4.0
15 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| S Mar 2, 2022 at 13:24 | history | suggested | Royi |
Add a missing tag
|
|
| Mar 2, 2022 at 9:27 | review | Suggested edits | |||
| S Mar 2, 2022 at 13:24 | |||||
| S Nov 29, 2021 at 3:03 | history | bounty ended | sparc_spread | ||
| S Nov 29, 2021 at 3:03 | history | notice removed | sparc_spread | ||
| Nov 28, 2021 at 12:44 | vote | accept | sparc_spread | ||
| Nov 27, 2021 at 1:59 | answer | added | Geoffrey Johnson | timeline score: 1 | |
| Nov 27, 2021 at 1:18 | history | edited | sparc_spread | CC BY-SA 4.0 |
added 136 characters in body
|
| Nov 27, 2021 at 1:17 | comment | added | sparc_spread | Yes, you are correct about $s$. But it can be calculated on any past data, the model is just predicting the input for calculating it. The question is which past data to use: training, test, or something else. | |
| Nov 26, 2021 at 20:18 | comment | added | Jonny Lomond | $s$ here must be the standard deviation of your metric, which is a function of the completed model, yes? $s$ will depend on your model, so the only hope of getting a sample size before the model is finished is to decide to bound the out-of-sample sd of your metric. | |
| Nov 25, 2021 at 5:04 | comment | added | sparc_spread | 1) Disparate input sets; 2) Train/test refers to the ML input. The significant std dev differentiations are in the past populations we've been analyzing for the power analysis. This std dev is itself highly variant... one month's worth of data can have a very different std dev from another's. | |
| Nov 22, 2021 at 22:19 | comment | added | B.Liu | Please can you clarify: 1) Whether the ML model in your treatment and the old source in control take the same or a disjoint input dataset to produce $\hat{x}$ and $x$ respectively? 2) Whether the train/test data refers to the input to the ML model, and why there is a significantly different sample standard deviation between these two sets of data? In general though, for experiments with multiple layers input/output that has its own level of variability, it often helps to perform power calculations using some simulations rather than relying on formulas. | |
| S Nov 22, 2021 at 21:48 | history | bounty started | sparc_spread | ||
| S Nov 22, 2021 at 21:48 | history | notice added | sparc_spread | Draw attention | |
| Nov 17, 2021 at 20:47 | history | edited | sparc_spread | CC BY-SA 4.0 |
added 5 characters in body
|
| Nov 17, 2021 at 18:07 | history | asked | sparc_spread | CC BY-SA 4.0 |