Timeline for Conducting multiple weighted t.tests to count over what fraction of a population a model performs significantly better or worse than another
Current License: CC BY-SA 4.0
6 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jul 10, 2022 at 21:34 | vote | accept | Palace Chan | ||
| Jul 10, 2022 at 21:33 | comment | added | Palace Chan | that’s a good point - i could aggregate the underlying statistics across the dimensions and obtain more of a “continuous comparison” vs the p.value approach is forcing an awkward layer of “discretization” with a cutoff… | |
| Jul 10, 2022 at 20:05 | history | edited | dipetkov | CC BY-SA 4.0 |
added 220 characters in body
|
| Jul 10, 2022 at 19:56 | comment | added | dipetkov | Across items -- assuming items are independent -- would be straightforward if you weren't trying to express better/same/worse as a p-value because you can just tabulate the proportion of better/same/worse. With a p-value, the conclusions about one item depends on the conclusion about all the other items (because of the necessity to adjust the p-values). | |
| Jul 10, 2022 at 18:55 | comment | added | Palace Chan |
agree on the insufficiency of correlation alone here, e.g. set.seed(0); x <- rnorm(1000); y <- x + rnorm(1000, sd=0.1); y2 <- 1+3*y; cor(y, y2); lm(x ~ y)$coefficients; lm(x ~ y2)$coefficients where $cor$ is same so need more to spot the scaling. One wants to know how often A/B performs better/worse/same. It is interesting to know this over time and across items. correlation is just one dimension (assuming other aspects are also checked) in which a better/worse "score" could be built and this approach with t-tests came to mind. Curious of course about better alternatives for "across items"
|
|
| Jul 10, 2022 at 18:24 | history | answered | dipetkov | CC BY-SA 4.0 |