Questions tagged [ab-test]
A/B testing, also known as split or bucket testing, is a controlled comparison of the effectiveness of variants of a website, email, or other commercial product.
427
questions
0
votes
0
answers
29
views
Validating power analysis
I'm currently running some experiments in web. The main idea is to validate the impact in some conversion metrics. The typical funnel we have is: Traffic -> Signup -> some Activation event.
Our ...
- 137
3
votes
1
answer
55
views
How does G*Power calculate sensitivity in z-test for two independent proportions
I am trying to recreate sensitivity calculation in Python by applying the same methodology as used in G*Power
z-tests > Proportions: Difference Between Two Independent Proportions > Sensitivity: ...
- 191
2
votes
1
answer
38
views
Multi-level modeling as a statistical criterion
I found Doordashe's article about conducting switchback experiments. In their article they use multi-level modeling instead of statistical criterions, which are usually used in A/B testing. They ...
- 23
2
votes
2
answers
88
views
Non-constant treatment periods for A/B test samples
I have 4 A/B testing samples with 4 treatment groups and their 4 respective control groups (n=10000~), with known (but not distinct) treatment periods. The issue is that the treatment periods overlap ...
0
votes
0
answers
68
views
Synthetic Control using CausalPy
I am using CausalPy (https://causalpy.readthedocs.io/en/latest/) to implement synthetic controls for Bayesian Geo-Lift.
Goal is to test a business initiative/feature (on website) in let's say 1 EU ...
0
votes
0
answers
34
views
When A/B testing a two sample hypothesis test of means, should we always use the welch t-test? [duplicate]
The Welch t-test is best used when we cannot make an equal variance assumption between our treatment and control groups (our two samples).
However, in A/B testing, it's not clear to me how we could ...
2
votes
1
answer
48
views
How to check if two segments of an A/B test have significant difference in the treatment effect
I am planning to launch an A/B test and there's a concern about a possible primacy/novelty effect. I've decided to segment users into two segments (New Users, Returning Users) and estimate if the ...
- 191
0
votes
0
answers
8
views
What is the right way to talk about user events in an AB experiment as random variables drawn from distinct distributions in a test?
Say we have a well setup A\B test that puts half of users in one arm, half in another. We are testing clicks on a website.
I'm trying to understand how to speak about the impressions as draws from a ...
0
votes
0
answers
73
views
Sample size calculation for AB testing - Non binomial ratio metric
I'm currently working on a sample size calculation for an upcoming AB test related to our mobile app. Up until now, I've been dealing with binomial metrics, such as the conversion rate, which is ...
- 101
0
votes
0
answers
43
views
Optimization metrics for a single-item recommendation system
I'm working on a recommendation system that's a bit different from ones I've built before. In particular, this system shows only the top item to the user, and the user can either click on it, dismiss ...
- 113
0
votes
0
answers
23
views
A T-test for difference of means and normality assumption [duplicate]
I am a little bit confused about normality assumption in T-Test for difference of two independent means. My reasoning now as follows:
If the distribution of the populations are non-normal, it's ok as ...
- 191
2
votes
1
answer
40
views
When failing to reject a null hypothesis can we also calculate the probability of observing the treatment assuming the alternative is true?
Say we have a null hypothesis, $H_0$ that there is no difference in the proportions between two arms of an experiment. You power the test to alpha and beta levels such that your $H_1$ MDE is 1% ...
0
votes
0
answers
29
views
How to design an A/B test when homogenous variants are not obvious
I would like to design an a/b test for menu for a group restaurants, which cater to online users, seeing if there is any impact on sales and orders. But the constraint is that we can only serve one ...
0
votes
1
answer
45
views
Sample size calculation (power analysis) when randomization unit and analysis unit are different
How to calculate sample size for online experiments when randomization and analysis unit are different?
For example, if we have an online learning platform for schools, and we make changes to make it ...
- 31
1
vote
1
answer
81
views
AB testing: Control was performing 0.5% better than experiment set before the initiation of experiment
So we introduced a new feature in our app, that would aid conversion (hypothetically).
When I tried to measure this incremental change in conversion, I split my base set of customers into control (C) ...
- 141
1
vote
0
answers
18
views
Which statistical test to use for customer buying subscription for A/B test
I'm running an A/B test, where control has default experience and the test variant has personalised experience.
The users I am targeting randomly are lapsed users on a particular subscription and ...
- 11
5
votes
1
answer
307
views
How do I know if I'm seeing Simpson's paradox?
I need some guidance on how to analyse the results of a subgroup breakdown in an A/B test.
I have the results of an (ongoing) A/B test and need to do an interim analysis on
The overall headline ...
- 115
1
vote
0
answers
15
views
Interpretation of Confidence Intervals for Business case
In frequentist hypothesis testing for AB tests, when we construct 95% confidence interval around the mean difference between version A and B, the mean is an unbiased estimate between A & B, and we ...
- 31
0
votes
0
answers
35
views
Converting a continuous metric to a proportion for increased power
I run online A/B tests and I guess I have some basic stats background, but I've encountered a situation where I don't feel confident about.
We plan to track page load time in each experiment to make ...
- 191
0
votes
1
answer
181
views
How to calculate MDE for proportions?
When conducting AB tests, we use power analysis to calculate sample size with alpha, power and MDE (minimal detectable effect) parameters.
Mean MDE for continuous variable seems intuitive: Using Cohen'...
- 31
0
votes
0
answers
21
views
Estimate the Lift impact on overall audience in A/B test
I am performing an A/B test analysis on two cohorts tests vs. control. I observed a positive lift of 1.5% (lift in new downloaders) between the test and control.
The total audience in my A/B test was ...
- 11
0
votes
0
answers
15
views
Calculate marginal CPA in Lift test
I want to measure the marginal Cost Per Accusation (CPA) for a treatment. The experiment is designed as follows:
We have two groups: control and treatment
For every group we measure cost and number ...
1
vote
1
answer
78
views
Measuring Advertisement Effect on Sales
I'm currently working on a project focusing on "Measuring the effect of an advertisement on sales of a product."
I am seeking advice as I encountered an intricate situation that requires ...
0
votes
0
answers
21
views
How I can establish A/B test framework?
I know that my question has big scope but I would like to get your perspective especially from statistical point of view.
I want to establish A/B framework specially for our feature tests. We are ...
- 101
1
vote
0
answers
37
views
Cuped for A/B tests with difference before the test
Can we use CUPED to get rid of the difference between groups, that existed before starting A/B test? Is it accurate?
- 11
0
votes
0
answers
69
views
Using regression predictions to evaluate randomised trials - good idea or not?
I need to evaluate the results of an experiment using a black-box ML model.
Normally, if I have a series of outcomes from the control group $ X_i $ and the test group $ Y_i $ I would just do a t-test ...
- 115
0
votes
1
answer
20
views
Odds Ratio or 2 Prop Z Test
We would like to conduct an AB Test between 2 groups of people and the apps that people have to see which ones over or underindex.
Which is more appropriate odds ratio or 2 proportional z test and why?...
1
vote
1
answer
79
views
In an A/B test, how can you check if assignment to the various buckets was truly random?
Trying to figure out how I can confirm that my A/B Test assignment is truly random. I know the runs test is used to test for randomness.
Is it possible to use the runs test to check if my A/B Test ...
- 23
1
vote
0
answers
47
views
How Can I Propagate Uncertainty in Sample Weights?
Background
An AB test is to be run on a webpage. Users are randomized to either treatment or control, and the main outcome, $y$, is a binary outcome. The risk of the outcome depends on day of the ...
- 33.5k
0
votes
0
answers
15
views
How many metrics can be tested in one AB test?
As part of the AB test, we had a design identical to the one on the screenshot, trying to direct users from the main page to the categories. Because search by query is not as converting and leads ...
1
vote
0
answers
37
views
Do I understand correctly how a split system should work in AB tests
I have a question about how a split system should work.
I have a product. Which looks like eBay.
We conducted an experiment that should increase the number of users who choose one of the categories.
...
0
votes
0
answers
72
views
Calculating statistical significance across differing metrics in A/B testing
First up, apologies that this probably a dumb question.
I am switching careers and just getting started in trying to understand statistical significance in A/B testing. I've come into issues quickly ...
- 101
1
vote
0
answers
45
views
How to model proportion data from an online experiment?
I have designed and run an online experiment in which we've slightly changed parts of a web page. Let's say users visit our website to place food orders and the order funnel looks like this: home --&...
- 1,014
1
vote
1
answer
81
views
Bootstrap hypothesis test and Power Analysis: should I build my Power Analysis on t-test or Bootstrap, if hp testing is done through Bootstrap?
I'm using the bootstrap percentile method to frequently do hypothesis testing (two-tailed test for independent sample where H0 marketing KPI of group A - marketing metric of group B = 0; H1 is ≠ 0) ...
2
votes
2
answers
148
views
Does it make sense to do the power analysis if one of the sample proportions is 0%? [closed]
Can we have any sample size if one of the sample proportions is 0. I mean even the slightest difference detected between the two groups would be because of the treatment in this case, and hence I do ...
- 21
0
votes
0
answers
56
views
How to calculate effect size to to use Power Analysis to get minimum sample size when standard deviation is not available (digital marketing AB test)?
TLDR: I can't find a way to calculate the effect size needed to be inputted to Power Analysis to get the minimum sample size needed to 90% statistical significance as I can't calculate standard ...
4
votes
2
answers
828
views
What if we can't find the difference in AB test?
What if we do tests, try reduce variance, but still can't get low p-value to reject the hypothesis. How to find out that there's no difference, and if so we can choose variant B just because we like ...
- 43
0
votes
1
answer
198
views
AB testing, which denominator to use for lower funnel metrics
Suppose I run AB testing for a website that does subscription business. The company offers a free trial for 7 days before automatically enrolling users to subscriptions and charging them unless they ...
- 1
0
votes
0
answers
49
views
Significance test for treatment when baseline of control and test groups are different
I have two groups of different sizes (control and test) and I want to measure the impacte of a certain treatment on the test group. Unfortunately, the control group has a different baseline compared ...
0
votes
0
answers
25
views
AB Test For Ecommerce Coupon Promotion
I'd like to run A/B Test for Ecommerce Coupon Promotion.
Random Sampling from Total User Group.
Random Grouping into Experimental(Give Coupon) / Control (Don't Give Coupon) 9:1 ratio
Situation likes ...
- 1
2
votes
1
answer
118
views
Calculating sample size when you have 2 treatments and uneven distribution
I have 2 new conversion webpages to test. Leadership recommends 80% traffic to existing conversion webpage and 10% each to the 2 new conversion webpages (non-negotiable)
How do I calculate sample size?...
- 21
0
votes
0
answers
26
views
Bayesian AB testing for Bernoulli Events with Time Patterns
I have read in this paper, about how to do Bayesian AB testing for conversions in digital advertising. One can start with uniform priors, and update them to Beta distributions based on ...
- 136
1
vote
1
answer
235
views
bootstrap best practice in AB testing
I want to compare a percentile metric for control vs treatment. Using bootstrap, I see two ways of doing so and wonder which one makes more sense.
Approach 1:
Every time:
Bootstrap the control group ...
- 93
0
votes
0
answers
34
views
Statistical inference in A/B testing: is it enough to compare observed test outcome to control distribution?
I want to run an A/B test to examine the effect of a marketing campaign on revenue. I’m using a synthetic control setup, hence I have to compare the revenue generated by treated units to the ...
- 1
2
votes
1
answer
200
views
A/B test with two metrics, we test one metrics for improvement on the condition the other one doesn't change
I have the following business request:
"Test this change on the landing page and implement it for all the visitors if the test shows improvement in the conversion rate, but if the page load speed ...
- 191
0
votes
0
answers
20
views
Best Power Analysis sample size calculator for multivariant AB Test
I am looking for recommendations for a sample size calculator for a multivariant (i.e. 5 variant) test. I was previously using t-tests to evaluate, but that seems to gravely overestimate sample size. ...
1
vote
0
answers
49
views
Bayesian AB Testing and Normal Distribution: How to Implement Posterior Variance Samples
I'm running an bayesian AB test to compare a normally distributed metric with unknown mean and variance between two groups. My goal is to compare the average value of the metric between the two groups,...
- 331
0
votes
1
answer
114
views
How to test the effect of price increase on subscription revenue in a mobile app?
I'm currently working on a mobile app that offers premium subscription plans. I'm interested in testing whether increasing the price of the subscription might result in a significant increase in ...
0
votes
1
answer
882
views
How to get Python Statsmodels to match Evan Miller's Famous AB Test Sample Size Calculator?
I'm trying to recreate this result. Which functions from statsmodels will do this?
Evan Miller Link: https://www.evanmiller.org/ab-testing/sample-size.html
I have tried the following:
...
- 3
2
votes
1
answer
81
views
Bootstrap in cluster experiments
I am planning an AB-test for something like a call-center. We are testing a new interface for the call-center operators. It has to be tested within a small group of roughly 20 operators. The main ...
- 21