# Questions tagged [ab-test]

A/B testing, also known as split or bucket testing, is a controlled comparison of the effectiveness of variants of a website, email, or other commercial product.

436 questions
Filter by
Sorted by
Tagged with
1 vote
47 views

### Impact of sample size on metric lift

Assume we ran three randomized user split AB experiments exp_1, exp_2, exp_3 one after the other. All three had the exact same treatments & control. Exp_1 was ran for 2 weeks during 06/01-06/14 . ...
• 43
1 vote
29 views

### simple random sampling and in-group comparison

We are conducting an A/B test on our hosting site where we sell four different plans (A, B, C, and D). Visitors to our site are randomly assigned to either a base UI (control) or a modified UI (...
• 201
56 views

### Experiment design to determine effect of ads on etsy shop performance

I am looking to design an experiment to determine the impact of turning on etsy ads for my shop. I opened a shop for plant pots and have made 27 sales in the 4 months I have been open. I am ...
• 41
30 views

### Impact of user allocation on AB experiment results

Will the ab test results change as the the number of exposed users change? Assuming , we have the ability to expose a certain % users to an AB experiment. For eg: we can specify that only 20% of the ...
• 43
1 vote
18 views

### Estimate sample variance of return-on-ad-spend (ROAS)?

I'm trying to make sense of the correct method to calculate the variance of ad-revenue for my marketing company. During an a/b test we need to estimate the sample variance to compute our confidence ...
65 views

### Interpreting by dividing up results of AB test

at my company we are doing AB tests (with 95% confidence) for features of our game (mobile app, hyper-casual game, Global scale). After the tests had ran its course, we have a practice of dividing up ...
29 views

### A/B Test for CTR without user-level data

Suppose I want to show two different ads and compare which one generates more clicks. By design I know that each user will see one of two ads at random every time s/he visits an external page where ...
• 21
1 vote
42 views

### Causal Inference: Meta Learners usage

I have been running causal inference using Econ ML package on my data. I have a dataset containing customers divided into treatment and control and many other features. I run matching on those and ...
1 vote
51 views

### Within group vs between group variance in an RCT: how to handle?

I have a question about how far the naive treament effect estimator can go wrong when there is appreciable within group variance vs between group variance in an RCT, and what we can do about it ...
• 149
1 vote
71 views

### Which statistical approach is best for diverse conversion rates in an A/B test?

Our software startup builds chat bots for ecommerce websites. The chatbot talks to customers that open the chat bot, and has the goal of closing the sale with the store’s main product. We have about ...
• 111
30 views

### Define metric to use after ab test is finished

Let's consider such a setup for an experiment: I've made some new feature to the existing system and want to test, whether this feature will bring positive value to the system. May be I don't know ...
• 111
1 vote
65 views

### How to proceed when the observed effect is smaller than the Minimum Detectable Effect?

Given an A/B test with an already defined sample size and variance, a power analysis was conducted to estimate the Minimum Detectable Effect (MDE) of a proportion. After the experiment was concluded, ...
29 views

### Regression models as goal metrics for experiments

What are the known issues or pitfalls associated with scoring regression/ML models on observations involved in a randomized experiment (a/b test, etc.)? In a randomized experiment, for most metrics (...
• 612
55 views

### A/B test: optimal split ratio for one control and multiple variants

I read in kohavi et al 2022 on section 7 that when you have multiple treatments in an A/B test, an even split (e.g., if 1 control and 3 treatments, allocate 25% to each) is not the best way to perform ...
• 133
1 vote
23 views

### Is mutual exclusivity important for an A/B test for an audience selection method?

Say I want to measure whether a set of business rules is better than random at identifying customers most likely to respond to an email. The steps are: Take the entire population of 200 people and ...
• 11
1 vote
47 views

### Can we do a reasonable statistics interpretation on A/B test if group size is not known

My coworker did A/B test finding a 6.5% diff in two groups conversion rates. First group has 1176 converted events and second group 1309 converted events. The total number of events before conversion ...
57 views

### Minimum sample size needed for A/B testing

Glossary: Sample SD = standard deviation of the samples that you currently have (control group). We havent started the treatment yet so we should not use the SD of the treatment group. MDE = "...
• 1,011
36 views

### Statistics of algorithm AB testing on occurrences only

I have AB testing in place and I have to compare the data but I have no clue which statistical test to use. I'll try to speak in dating app terms because I think it makes it easier to understand. I ...
1 vote
15 views

### What implications does the discrepancy between the observed Minimum Detectable Effect and the initial MDE estimation have on the interpretation

Suppose the application has a click rate of 5%. A new version may improve this. Suppose that frequentists approach is used. To estimate the sample size the click rate of the new version is estimate ...
• 111
45 views

### Validating power analysis

I'm currently running some experiments in web. The main idea is to validate the impact in some conversion metrics. The typical funnel we have is: Traffic -> Signup -> some Activation event. Our ...
• 187
73 views

### How does G*Power calculate sensitivity in z-test for two independent proportions

I am trying to recreate sensitivity calculation in Python by applying the same methodology as used in G*Power z-tests > Proportions: Difference Between Two Independent Proportions > Sensitivity: ...
• 213
46 views

### Multi-level modeling as a statistical criterion

I found Doordashe's article about conducting switchback experiments. In their article they use multi-level modeling instead of statistical criterions, which are usually used in A/B testing. They ...
98 views

### Non-constant treatment periods for A/B test samples

I have 4 A/B testing samples with 4 treatment groups and their 4 respective control groups (n=10000~), with known (but not distinct) treatment periods. The issue is that the treatment periods overlap ...
137 views

### Synthetic Control using CausalPy

I am using CausalPy (https://causalpy.readthedocs.io/en/latest/) to implement synthetic controls for Bayesian Geo-Lift. Goal is to test a business initiative/feature (on website) in let's say 1 EU ...
1 vote
85 views

### When A/B testing a two sample hypothesis test of means, should we always use the welch t-test? [duplicate]

The Welch t-test is best used when we cannot make an equal variance assumption between our treatment and control groups (our two samples). However, in A/B testing, it's not clear to me how we could ...
84 views

### How to check if two segments of an A/B test have significant difference in the treatment effect

I am planning to launch an A/B test and there's a concern about a possible primacy/novelty effect. I've decided to segment users into two segments (New Users, Returning Users) and estimate if the ...
• 213
9 views

### What is the right way to talk about user events in an AB experiment as random variables drawn from distinct distributions in a test?

Say we have a well setup A\B test that puts half of users in one arm, half in another. We are testing clicks on a website. I'm trying to understand how to speak about the impressions as draws from a ...
148 views

### Sample size calculation for AB testing - Non binomial ratio metric

I'm currently working on a sample size calculation for an upcoming AB test related to our mobile app. Up until now, I've been dealing with binomial metrics, such as the conversion rate, which is ...
• 101
46 views

### Optimization metrics for a single-item recommendation system

I'm working on a recommendation system that's a bit different from ones I've built before. In particular, this system shows only the top item to the user, and the user can either click on it, dismiss ...
• 113
24 views

### A T-test for difference of means and normality assumption [duplicate]

I am a little bit confused about normality assumption in T-Test for difference of two independent means. My reasoning now as follows: If the distribution of the populations are non-normal, it's ok as ...
• 213
50 views

### When failing to reject a null hypothesis can we also calculate the probability of observing the treatment assuming the alternative is true?

Say we have a null hypothesis, $H_0$ that there is no difference in the proportions between two arms of an experiment. You power the test to alpha and beta levels such that your $H_1$ MDE is 1% ...
34 views

### How to design an A/B test when homogenous variants are not obvious

I would like to design an a/b test for menu for a group restaurants, which cater to online users, seeing if there is any impact on sales and orders. But the constraint is that we can only serve one ...
92 views

### Sample size calculation (power analysis) when randomization unit and analysis unit are different

How to calculate sample size for online experiments when randomization and analysis unit are different? For example, if we have an online learning platform for schools, and we make changes to make it ...
1 vote
107 views

### AB testing: Control was performing 0.5% better than experiment set before the initiation of experiment

So we introduced a new feature in our app, that would aid conversion (hypothetically). When I tried to measure this incremental change in conversion, I split my base set of customers into control (C) ...
• 141
1 vote
23 views

### Which statistical test to use for customer buying subscription for A/B test

I'm running an A/B test, where control has default experience and the test variant has personalised experience. The users I am targeting randomly are lapsed users on a particular subscription and ...
406 views

### How do I know if I'm seeing Simpson's paradox?

I need some guidance on how to analyse the results of a subgroup breakdown in an A/B test. I have the results of an (ongoing) A/B test and need to do an interim analysis on The overall headline ...
• 149
1 vote
15 views

### Interpretation of Confidence Intervals for Business case

In frequentist hypothesis testing for AB tests, when we construct 95% confidence interval around the mean difference between version A and B, the mean is an unbiased estimate between A & B, and we ...
54 views

### Converting a continuous metric to a proportion for increased power

I run online A/B tests and I guess I have some basic stats background, but I've encountered a situation where I don't feel confident about. We plan to track page load time in each experiment to make ...
• 213
748 views

### How to calculate MDE for proportions?

When conducting AB tests, we use power analysis to calculate sample size with alpha, power and MDE (minimal detectable effect) parameters. Mean MDE for continuous variable seems intuitive: Using Cohen'...
28 views

### Estimate the Lift impact on overall audience in A/B test

I am performing an A/B test analysis on two cohorts tests vs. control. I observed a positive lift of 1.5% (lift in new downloaders) between the test and control. The total audience in my A/B test was ...
21 views

### Calculate marginal CPA in Lift test

I want to measure the marginal Cost Per Accusation (CPA) for a treatment. The experiment is designed as follows: We have two groups: control and treatment For every group we measure cost and number ...
93 views

I'm currently working on a project focusing on "Measuring the effect of an advertisement on sales of a product." I am seeking advice as I encountered an intricate situation that requires ...
24 views

### How I can establish A/B test framework?

I know that my question has big scope but I would like to get your perspective especially from statistical point of view. I want to establish A/B framework specially for our feature tests. We are ...
• 101
1 vote
49 views

### Cuped for A/B tests with difference before the test

Can we use CUPED to get rid of the difference between groups, that existed before starting A/B test? Is it accurate?
• 11
72 views

### Using regression predictions to evaluate randomised trials - good idea or not?

I need to evaluate the results of an experiment using a black-box ML model. Normally, if I have a series of outcomes from the control group $X_i$ and the test group $Y_i$ I would just do a t-test ...
• 149
44 views

### Odds Ratio or 2 Prop Z Test

We would like to conduct an AB Test between 2 groups of people and the apps that people have to see which ones over or underindex. Which is more appropriate odds ratio or 2 proportional z test and why?...
1 vote
232 views

### In an A/B test, how can you check if assignment to the various buckets was truly random?

Trying to figure out how I can confirm that my A/B Test assignment is truly random. I know the runs test is used to test for randomness. Is it possible to use the runs test to check if my A/B Test ...
• 65
1 vote
54 views

### How Can I Propagate Uncertainty in Sample Weights?

Background An AB test is to be run on a webpage. Users are randomized to either treatment or control, and the main outcome, $y$, is a binary outcome. The risk of the outcome depends on day of the ...
1 vote