Timeline for A/B testing ratio of sums
Current License: CC BY-SA 4.0
32 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Sep 7, 2022 at 14:24 | comment | added | Sextus Empiricus | @getup8 this equation 5 from the towards datascience blog is the same as the equation used here. Small differences are in details like computing the variance instead of the standard deviation. | |
| Sep 7, 2022 at 14:18 | comment | added | Xavier Bourret Sicotte | Yes, the exact formula, its approximation with the delta method, and bootstrapping all yielded nearly identical results on our real AB test data, so I would go with whatever solution you are most comfortable with. The bootstrap was too slow and resource intensive given the dataset size, but the delta method worked fine. The conclusion for us was the the true variance was actually smaller (using delta) than the variance we had using the z-test, so our tests were more powerful than we thought | |
| Sep 6, 2022 at 19:05 | comment | added | getup8 | Upon looking at this more though, I think these different references are all doing the same thing, just with nuanced differences in exactly what they're calculating. | |
| Sep 6, 2022 at 18:54 | comment | added | getup8 | @SextusEmpiricus yeah this is a blog post I was comparing this to: medium.com/@ahmadnuraziz3/… and related, towardsdatascience.com/… | |
| Sep 6, 2022 at 5:44 | comment | added | Sextus Empiricus | @getup8, can you give a reference for the slightly different formula's. | |
| Sep 6, 2022 at 4:25 | comment | added | getup8 | @XavierBourretSicotte (or anyone else), I'm having a hard time actually doing this end-to-end (the Delta method) for an a/b test. The formula given seems slightly different than others I've seen (it's not normalized by N?) and it'd be great to see what do actually do with the result (sorry, slight noob). | |
| Mar 25, 2021 at 9:28 | comment | added | Xi'an | Typo: I meant Marsaglia (1965). | |
| Mar 25, 2021 at 9:16 | history | edited | Sextus Empiricus | CC BY-SA 4.0 |
added 297 characters in body
|
| Mar 25, 2021 at 9:03 | comment | added | Sextus Empiricus | @Xi'an That's a great reference. I didn't check it when reading Hinkley. | |
| Mar 25, 2021 at 8:14 | comment | added | Xi'an | As a minor remark (on a great answer), George Marsaglia had an earlier publication(1969) on the ratio density, as “Ratios of Normal Variables and Ratios of Sums of Uniform Variables.” Journal of the American Statistical Association, 60, 193–204. Hinkley (1969) wrongly pointed out a mistake in that paper, corrected in Hinley (1970 Biometrika). | |
| Sep 2, 2020 at 8:06 | history | edited | Sextus Empiricus | CC BY-SA 4.0 |
added 11 characters in body
|
| S Sep 2, 2020 at 8:04 | history | suggested | CommunityBot | CC BY-SA 4.0 |
Plugging $a(z) = \mu_x / \mu_y yields the covariance in the middle term since you write \rho = cov(X,Y) / (\sigma_x * \sigma_y).
|
| Sep 2, 2020 at 7:19 | review | Suggested edits | |||
| S Sep 2, 2020 at 8:04 | |||||
| May 16, 2019 at 16:13 | comment | added | Sextus Empiricus | You can observe this by plotting the results from the bootstrapping. I imagine you end up with the same mean for the sum of star items and total item, but with different variance (shuffling a lot single transactions gives you smaller variance than shuffling a few large packages of transactions). | |
| May 16, 2019 at 16:01 | comment | added | Sextus Empiricus | In the example that I gave, with the two by two table, it gives a distribution for the users placed into four different classes. The underlying transactions might be different and you may not get the same distribution for users by sampling the distribution of transactions (E.g. by using a compound distribution). | |
| May 16, 2019 at 15:56 | comment | added | Sextus Empiricus | The reason for the difference is that the transactions are not independently distributed among the users. For instance you may have groups of users that buy relatively more or less in total or more or less of star items. | |
| May 16, 2019 at 15:44 | comment | added | Xavier Bourret Sicotte | Indeed - i dont see why they should be the same - but bootstrapping at user level makes intuitive sense because that is the randomization unit of the ab test... | |
| May 16, 2019 at 15:37 | comment | added | Sextus Empiricus | @XavierBourretSicotte I would have to refresh my memory about this problem and look up what the transaction level and user level entails. But, I imagine they won't be the same (why should they?). | |
| May 16, 2019 at 15:14 | comment | added | Xavier Bourret Sicotte | @MartijnWeterings i confirm that using the real data set and metric and bootstrapping at transaction level gives a result that follows exactly the hinkley distribution - so again +1 for your answer! Binomial assumption gives the same mean but the variance is overestimated by a factor of 5 - my new problem now is that bootstrap at the user level gives a totally different mean... Any ideas? | |
| May 15, 2019 at 18:11 | history | edited | Xavier Bourret Sicotte | CC BY-SA 4.0 |
Hi - just added a missing a(z)^2 in the denominator of the last part of f(z) - taken from the Hinkley 1969 paper
|
| Mar 29, 2019 at 16:56 | history | edited | Sextus Empiricus | CC BY-SA 4.0 |
error in plotting the histograms
|
| Mar 29, 2019 at 16:35 | vote | accept | Xavier Bourret Sicotte | ||
| Mar 29, 2019 at 16:35 | history | bounty ended | Xavier Bourret Sicotte | ||
| Mar 29, 2019 at 9:34 | comment | added | Sextus Empiricus | @usεr11852 The distribution of the sales from single users does not need to be normal. Like in the example where it is some sort of multivariate Bernoulli distribution. What you do need is that the sample distribution of the sum of sales of users needs to be (approximately) normal. This will be the case when the distribution for single users has finite moments (then they will vanish for the sum). They will indeed vanish faster when you do not have very high skew or fat tails, and you will be needing a sufficient sample size (but the same is true for bootstrapping). | |
| Mar 29, 2019 at 7:05 | comment | added | usεr11852 | (+1) Just a comment: have you checked what happens if the underlying distribution of the metric is not normal but instead it is fat-tailed (e.g. log-normal)? To that extent, quickly seeing this, the distribution of the ratio never has values of $1$ (or close to that) while it is "certain" that some people will buy only one (or two) starred items as a whole. To quote Hole (2007): "The delta method is the most accurate when the data is well-conditioned, while the bootstrap is more robust to noisy data and misspecification of the model.". | |
| Mar 28, 2019 at 20:30 | comment | added | Xavier Bourret Sicotte | Lets continue this discussion in chat ? chat.stackexchange.com/rooms/91683/ab-test-ratio-of-sums thanks ! | |
| Mar 28, 2019 at 20:23 | comment | added | Sextus Empiricus | The Delta method also includes correlation. Hinkley is only more accurate but the difference is not so important when the variance of your variables is small. You could say that$$x/y-\mu_x/\mu_y\approx(x-\mu_x)\frac{1}{\mu_y}+(y-\mu_y)\frac{\mu_x}{\mu_y^2}$$ie you approximate the ratio error by a linear sum for which the variance can be easily computed (also with correlation). When the distribution spreads out over a larger area then those iso-lines are not any more approximately equidistant and parallel, then the difference between the Delta method and Hinkley's formula becomes larger. | |
| Mar 28, 2019 at 19:39 | comment | added | Xavier Bourret Sicotte | Fantastic answer Martijn, as always. So to recap, once I use the Delta method approximation, or the more accurate Hinkley result which includes correlation, I obtain a new r.v. which is asymptotically normal. I can then perform a standard z-test of difference in means on this new r.v. for control and variant ? i.e. H0 be both control and variant have the same mean, or mean = 0 ? I have actually never seen an example of using the delta method for a test of difference of means... do you have an example ? | |
| Mar 28, 2019 at 19:21 | history | edited | Sextus Empiricus | CC BY-SA 4.0 |
edited body
|
| Mar 28, 2019 at 19:15 | history | edited | Sextus Empiricus | CC BY-SA 4.0 |
added 3 characters in body
|
| Mar 28, 2019 at 17:15 | history | edited | Sextus Empiricus | CC BY-SA 4.0 |
added 44 characters in body
|
| Mar 28, 2019 at 17:07 | history | answered | Sextus Empiricus | CC BY-SA 4.0 |