Skip to main content

Questions tagged [ab-test]

A/B testing, also known as split or bucket testing, is a controlled comparison of the effectiveness of variants of a website, email, or other commercial product.

Filter by
Sorted by
Tagged with
1 vote
2 answers
47 views

Impact of sample size on metric lift

Assume we ran three randomized user split AB experiments exp_1, exp_2, exp_3 one after the other. All three had the exact same treatments & control. Exp_1 was ran for 2 weeks during 06/01-06/14 . ...
bp0308's user avatar
  • 43
1 vote
1 answer
29 views

simple random sampling and in-group comparison

We are conducting an A/B test on our hosting site where we sell four different plans (A, B, C, and D). Visitors to our site are randomly assigned to either a base UI (control) or a modified UI (...
Iman's user avatar
  • 201
4 votes
1 answer
56 views

Experiment design to determine effect of ads on etsy shop performance

I am looking to design an experiment to determine the impact of turning on etsy ads for my shop. I opened a shop for plant pots and have made 27 sales in the 4 months I have been open. I am ...
Nick G's user avatar
  • 41
0 votes
0 answers
30 views

Impact of user allocation on AB experiment results

Will the ab test results change as the the number of exposed users change? Assuming , we have the ability to expose a certain % users to an AB experiment. For eg: we can specify that only 20% of the ...
bp0308's user avatar
  • 43
1 vote
1 answer
18 views

Estimate sample variance of return-on-ad-spend (ROAS)?

I'm trying to make sense of the correct method to calculate the variance of ad-revenue for my marketing company. During an a/b test we need to estimate the sample variance to compute our confidence ...
Zach Cuddihy's user avatar
0 votes
0 answers
65 views

Interpreting by dividing up results of AB test

at my company we are doing AB tests (with 95% confidence) for features of our game (mobile app, hyper-casual game, Global scale). After the tests had ran its course, we have a practice of dividing up ...
Thinh Vuong's user avatar
2 votes
0 answers
29 views

A/B Test for CTR without user-level data

Suppose I want to show two different ads and compare which one generates more clicks. By design I know that each user will see one of two ads at random every time s/he visits an external page where ...
ssrg's user avatar
  • 21
1 vote
1 answer
42 views

Causal Inference: Meta Learners usage

I have been running causal inference using Econ ML package on my data. I have a dataset containing customers divided into treatment and control and many other features. I run matching on those and ...
Marco Miglionico's user avatar
1 vote
1 answer
51 views

Within group vs between group variance in an RCT: how to handle?

I have a question about how far the naive treament effect estimator can go wrong when there is appreciable within group variance vs between group variance in an RCT, and what we can do about it ...
Tom Kealy's user avatar
  • 149
1 vote
1 answer
71 views

Which statistical approach is best for diverse conversion rates in an A/B test?

Our software startup builds chat bots for ecommerce websites. The chatbot talks to customers that open the chat bot, and has the goal of closing the sale with the store’s main product. We have about ...
Rage's user avatar
  • 111
0 votes
2 answers
30 views

Define metric to use after ab test is finished

Let's consider such a setup for an experiment: I've made some new feature to the existing system and want to test, whether this feature will bring positive value to the system. May be I don't know ...
Joitandr's user avatar
  • 111
1 vote
0 answers
65 views

How to proceed when the observed effect is smaller than the Minimum Detectable Effect?

Given an A/B test with an already defined sample size and variance, a power analysis was conducted to estimate the Minimum Detectable Effect (MDE) of a proportion. After the experiment was concluded, ...
loglossbb's user avatar
3 votes
0 answers
29 views

Regression models as goal metrics for experiments

What are the known issues or pitfalls associated with scoring regression/ML models on observations involved in a randomized experiment (a/b test, etc.)? In a randomized experiment, for most metrics (...
user1993951's user avatar
2 votes
1 answer
55 views

A/B test: optimal split ratio for one control and multiple variants

I read in kohavi et al 2022 on section 7 that when you have multiple treatments in an A/B test, an even split (e.g., if 1 control and 3 treatments, allocate 25% to each) is not the best way to perform ...
user11513145's user avatar
1 vote
1 answer
23 views

Is mutual exclusivity important for an A/B test for an audience selection method?

Say I want to measure whether a set of business rules is better than random at identifying customers most likely to respond to an email. The steps are: Take the entire population of 200 people and ...
djs's user avatar
  • 11
1 vote
1 answer
47 views

Can we do a reasonable statistics interpretation on A/B test if group size is not known

My coworker did A/B test finding a 6.5% diff in two groups conversion rates. First group has 1176 converted events and second group 1309 converted events. The total number of events before conversion ...
Antti Hemilä's user avatar
2 votes
0 answers
57 views

Minimum sample size needed for A/B testing

Glossary: Sample SD = standard deviation of the samples that you currently have (control group). We havent started the treatment yet so we should not use the SD of the treatment group. MDE = "...
Katsu's user avatar
  • 1,011
0 votes
0 answers
36 views

Statistics of algorithm AB testing on occurrences only

I have AB testing in place and I have to compare the data but I have no clue which statistical test to use. I'll try to speak in dating app terms because I think it makes it easier to understand. I ...
sparrow-cake-lizard-web's user avatar
1 vote
0 answers
15 views

What implications does the discrepancy between the observed Minimum Detectable Effect and the initial MDE estimation have on the interpretation

Suppose the application has a click rate of 5%. A new version may improve this. Suppose that frequentists approach is used. To estimate the sample size the click rate of the new version is estimate ...
rfalke's user avatar
  • 111
2 votes
0 answers
45 views

Validating power analysis

I'm currently running some experiments in web. The main idea is to validate the impact in some conversion metrics. The typical funnel we have is: Traffic -> Signup -> some Activation event. Our ...
marz's user avatar
  • 187
4 votes
1 answer
73 views

How does G*Power calculate sensitivity in z-test for two independent proportions

I am trying to recreate sensitivity calculation in Python by applying the same methodology as used in G*Power z-tests > Proportions: Difference Between Two Independent Proportions > Sensitivity: ...
Eugene Krall's user avatar
2 votes
1 answer
46 views

Multi-level modeling as a statistical criterion

I found Doordashe's article about conducting switchback experiments. In their article they use multi-level modeling instead of statistical criterions, which are usually used in A/B testing. They ...
Соня Бондарь's user avatar
2 votes
2 answers
98 views

Non-constant treatment periods for A/B test samples

I have 4 A/B testing samples with 4 treatment groups and their 4 respective control groups (n=10000~), with known (but not distinct) treatment periods. The issue is that the treatment periods overlap ...
scooptywoopty's user avatar
0 votes
0 answers
137 views

Synthetic Control using CausalPy

I am using CausalPy (https://causalpy.readthedocs.io/en/latest/) to implement synthetic controls for Bayesian Geo-Lift. Goal is to test a business initiative/feature (on website) in let's say 1 EU ...
Siddhartha Srivastava's user avatar
1 vote
0 answers
85 views

When A/B testing a two sample hypothesis test of means, should we always use the welch t-test? [duplicate]

The Welch t-test is best used when we cannot make an equal variance assumption between our treatment and control groups (our two samples). However, in A/B testing, it's not clear to me how we could ...
Estimate the estimators's user avatar
2 votes
1 answer
84 views

How to check if two segments of an A/B test have significant difference in the treatment effect

I am planning to launch an A/B test and there's a concern about a possible primacy/novelty effect. I've decided to segment users into two segments (New Users, Returning Users) and estimate if the ...
Eugene Krall's user avatar
0 votes
0 answers
9 views

What is the right way to talk about user events in an AB experiment as random variables drawn from distinct distributions in a test?

Say we have a well setup A\B test that puts half of users in one arm, half in another. We are testing clicks on a website. I'm trying to understand how to speak about the impressions as draws from a ...
Estimate the estimators's user avatar
0 votes
0 answers
148 views

Sample size calculation for AB testing - Non binomial ratio metric

I'm currently working on a sample size calculation for an upcoming AB test related to our mobile app. Up until now, I've been dealing with binomial metrics, such as the conversion rate, which is ...
Discipulus's user avatar
0 votes
0 answers
46 views

Optimization metrics for a single-item recommendation system

I'm working on a recommendation system that's a bit different from ones I've built before. In particular, this system shows only the top item to the user, and the user can either click on it, dismiss ...
Kyranstar's user avatar
  • 113
0 votes
0 answers
24 views

A T-test for difference of means and normality assumption [duplicate]

I am a little bit confused about normality assumption in T-Test for difference of two independent means. My reasoning now as follows: If the distribution of the populations are non-normal, it's ok as ...
Eugene Krall's user avatar
2 votes
1 answer
50 views

When failing to reject a null hypothesis can we also calculate the probability of observing the treatment assuming the alternative is true?

Say we have a null hypothesis, $H_0$ that there is no difference in the proportions between two arms of an experiment. You power the test to alpha and beta levels such that your $H_1$ MDE is 1% ...
Estimate the estimators's user avatar
0 votes
0 answers
34 views

How to design an A/B test when homogenous variants are not obvious

I would like to design an a/b test for menu for a group restaurants, which cater to online users, seeing if there is any impact on sales and orders. But the constraint is that we can only serve one ...
fierystan's user avatar
0 votes
1 answer
92 views

Sample size calculation (power analysis) when randomization unit and analysis unit are different

How to calculate sample size for online experiments when randomization and analysis unit are different? For example, if we have an online learning platform for schools, and we make changes to make it ...
user1456579's user avatar
1 vote
1 answer
107 views

AB testing: Control was performing 0.5% better than experiment set before the initiation of experiment

So we introduced a new feature in our app, that would aid conversion (hypothetically). When I tried to measure this incremental change in conversion, I split my base set of customers into control (C) ...
Narahari B M's user avatar
1 vote
0 answers
23 views

Which statistical test to use for customer buying subscription for A/B test

I'm running an A/B test, where control has default experience and the test variant has personalised experience. The users I am targeting randomly are lapsed users on a particular subscription and ...
rogercake's user avatar
5 votes
1 answer
406 views

How do I know if I'm seeing Simpson's paradox?

I need some guidance on how to analyse the results of a subgroup breakdown in an A/B test. I have the results of an (ongoing) A/B test and need to do an interim analysis on The overall headline ...
Tom Kealy's user avatar
  • 149
1 vote
0 answers
15 views

Interpretation of Confidence Intervals for Business case

In frequentist hypothesis testing for AB tests, when we construct 95% confidence interval around the mean difference between version A and B, the mean is an unbiased estimate between A & B, and we ...
user1456579's user avatar
0 votes
0 answers
54 views

Converting a continuous metric to a proportion for increased power

I run online A/B tests and I guess I have some basic stats background, but I've encountered a situation where I don't feel confident about. We plan to track page load time in each experiment to make ...
Eugene Krall's user avatar
0 votes
1 answer
748 views

How to calculate MDE for proportions?

When conducting AB tests, we use power analysis to calculate sample size with alpha, power and MDE (minimal detectable effect) parameters. Mean MDE for continuous variable seems intuitive: Using Cohen'...
user1456579's user avatar
0 votes
0 answers
28 views

Estimate the Lift impact on overall audience in A/B test

I am performing an A/B test analysis on two cohorts tests vs. control. I observed a positive lift of 1.5% (lift in new downloaders) between the test and control. The total audience in my A/B test was ...
rogercake's user avatar
0 votes
0 answers
21 views

Calculate marginal CPA in Lift test

I want to measure the marginal Cost Per Accusation (CPA) for a treatment. The experiment is designed as follows: We have two groups: control and treatment For every group we measure cost and number ...
Wesam Adel's user avatar
2 votes
1 answer
93 views

Measuring Advertisement Effect on Sales

I'm currently working on a project focusing on "Measuring the effect of an advertisement on sales of a product." I am seeking advice as I encountered an intricate situation that requires ...
Aniruddha Mitra's user avatar
0 votes
0 answers
24 views

How I can establish A/B test framework?

I know that my question has big scope but I would like to get your perspective especially from statistical point of view. I want to establish A/B framework specially for our feature tests. We are ...
Ephesus's user avatar
  • 101
1 vote
0 answers
49 views

Cuped for A/B tests with difference before the test

Can we use CUPED to get rid of the difference between groups, that existed before starting A/B test? Is it accurate?
Samantha 's user avatar
0 votes
0 answers
72 views

Using regression predictions to evaluate randomised trials - good idea or not?

I need to evaluate the results of an experiment using a black-box ML model. Normally, if I have a series of outcomes from the control group $ X_i $ and the test group $ Y_i $ I would just do a t-test ...
Tom Kealy's user avatar
  • 149
0 votes
1 answer
44 views

Odds Ratio or 2 Prop Z Test

We would like to conduct an AB Test between 2 groups of people and the apps that people have to see which ones over or underindex. Which is more appropriate odds ratio or 2 proportional z test and why?...
Hider1466's user avatar
1 vote
1 answer
232 views

In an A/B test, how can you check if assignment to the various buckets was truly random?

Trying to figure out how I can confirm that my A/B Test assignment is truly random. I know the runs test is used to test for randomness. Is it possible to use the runs test to check if my A/B Test ...
ibarbo's user avatar
  • 65
1 vote
0 answers
54 views

How Can I Propagate Uncertainty in Sample Weights?

Background An AB test is to be run on a webpage. Users are randomized to either treatment or control, and the main outcome, $y$, is a binary outcome. The risk of the outcome depends on day of the ...
Demetri Pananos's user avatar
1 vote
0 answers
39 views

Do I understand correctly how a split system should work in AB tests

I have a question about how a split system should work. I have a product. Which looks like eBay. We conducted an experiment that should increase the number of users who choose one of the categories. ...
Roman Stasiuk's user avatar
1 vote
0 answers
49 views

How to model proportion data from an online experiment?

I have designed and run an online experiment in which we've slightly changed parts of a web page. Let's say users visit our website to place food orders and the order funnel looks like this: home --&...
Adrià Luz's user avatar
  • 1,034

1
2 3 4 5
9