Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [ab-test]

A/B testing, also known as split or bucket testing, is a controlled comparison of the effectiveness of variants of a website, email, or other commercial product.

0
votes
0answers
8 views

Why we have to use pooled sample proportion to calculate the pooled standard deviation of the difference?

The context can be found in Hypothesis Test: Difference Between Proportions It says there: ...
1
vote
1answer
17 views

Need help determining sample size for an AB test with two metrics

I need to run an experiment with customer support agents who are rated on two metrics by the customer: average issue resolution rate (proportion of issues resolved) and average star rating (1-5). ...
-1
votes
0answers
19 views

Evaluating results from control group testing

In the classic case of target and control groups, the target and control group receive a differential treatment and the effect of this treatment is evaluated. This assumes that every member within the ...
1
vote
0answers
28 views

Bayesian A/B testing and error estimates

Someone from my work started using an AB calculator that they found online and asked my take on the approach which i assume is incorrect. The approach takes the conversion and visits for control and ...
1
vote
0answers
16 views

Measuring incremental revenue / uplift

I run a campaign with an experimental design setup. The target and control group is representative and split 70/30. The target group receives an email communication The dynamics of the campaign ...
0
votes
1answer
22 views

What is necessary sample size for B in A/B test with unequal sample sizes?

I have a situation in which I want to measure if the error rate in system B (test) is worse than the error rate in system A (control). System A has a sample size of 100 and that cannot be modified. So ...
3
votes
3answers
540 views

2 sample t test for sample sizes - 30,000 and 150,000

I have 2 samples, one with sample size of 30,000 customers and the other with 150,000. I have to perform a 2 sample t test(on conversion rates of the 2 groups). My question is, will t test in this ...
0
votes
0answers
20 views

Interpretation for Chi-Square Test for Independence in A/B/n testing

I am testing the results of a marketing campaign which involves randomly splitting a group of people into 4 treatments. One group acts as a control and is shown no advertising, the other 3 groups are ...
1
vote
0answers
15 views

AB Test for Total Revenue: How to take both of conversion rate and average order value into account in my ab-test?

I have some purchase data for two groups of people, for instance: Method Revenue ------ ------- A 12 A 0 A 0 A 4 A 0 B 0 B 8 B ...
1
vote
1answer
33 views

AB testing and hypothesis testing

I have an entire market for which I plan to run two offers. I randomly split the market 50/50. All of Group A receives offer 1 and all of group B receives offer 2. I want to compare proportions of ...
1
vote
0answers
36 views

A/B testing for binary classification tasks

I work on project where each user has an IoT sensor and an ML algorithm tries to identify a given situation. The user carries the sensor 24/7 but I split the time into 'sessions'. For each session, I ...
0
votes
0answers
22 views

Confidence interval in Chi squared testing

How to calculate confidence interval when performing a Chi-Square significance testing? For reference where it's used - http://www.evanmiller.org/ab-testing/chi-squared.html Can we apply the same ...
8
votes
4answers
490 views

A/B testing ratio of sums

Context Consider the following scenario for a company selling goods online. A user can purchase several items (i.e. basket of items), some of which are of particular importance and are tracked ...
0
votes
0answers
27 views

How to calculate significance for an A/B test for average revenue per daily active user over a 1-week test period?

One simple approach is to use "per active day" metrics. e.g. "revenue_per_active_day" . However there is a fundamental flaw with this approach. e.g. during a 7-day period, let's say user 1 is active ...
2
votes
1answer
66 views

AB Testing vs Hypothesis Testing

I'm having trouble making the full connection between AB Testing and Hypothesis testing. Hypothesis Testing Conducting a test to compare two means, we need the standard deviation of both samples, and ...
2
votes
2answers
51 views

statistical significance of A B test if some observation are removed due to errors

I have ran an A/B test and using chi-square test of independence, results shows that group A improves on group B by less than 5% (statistically significant) YAY!!! Unfortunately, after few checks I ...
4
votes
1answer
95 views

AB test sample size calculation by hand

Evan Miller has created a well-known online AB test sample size calculator. For the sake of being able to program and modify this formula, I would like to know how to calculate sample size Evan Miller-...
0
votes
1answer
19 views

A\B Testing with Sorting

In my company while splitting site users for AB test we do the following: we choose parameters that seem the most important for us (for example: gender, life time value, average check, etc), if they ...
0
votes
1answer
24 views

how to present results for two group success rate with confidence interval

I have two groups A and B, and for each data point I have two possible outcomes, success and failures. For each group, I have calculated the success rate, and the confidence interval for a binomial ...
0
votes
0answers
4 views

Statistics of Diff in diff with optimal group

I am trying to run a diff in diff to measure the impact of a feature. I want to run the experiment per geo region and I have 90 regions. To run a proper diff in diff I should split in have the ...
0
votes
0answers
13 views

Conversion/Revenue Attribution in AB experiments

I have a question on A/B testing attribution. I have a test running on mobile app and website at the same time. Unfortunately the way the test is designed ...
0
votes
1answer
21 views

In AB test do the sample size calculations represent the minimum or maximum size required for measuring difference between the test and control

A quick question on the frequentest approach of sample size calculation. I recently came across this below note: The sample size required should be strictly enforced; if the power analysis shows ...
0
votes
0answers
20 views

A/B test : Is there a way interpreting factor “A” group is greater than factor “B” group?

Plz.. understand my lack of English skills.. T.T I have a problem dealing with statistical hypothesis testing. To be specific, we have two special delivery method like (new) : method "A", (old) : ...
1
vote
0answers
35 views

Good numbers, but something feels wrong

This is more a qualitative question than a quantitive one. I have done an A B test to try to increase use of a particular feature (feature 1) by increasing discoverability. I measure it by seeing ...
0
votes
2answers
138 views

Multi-armed bandit vs AB testing

I'm trying to understand the difference between AB Testing and Multi armed bandit and when we should use one of the other. It seems to me that Multi-armed bandit algorithms are statistically valid and ...
0
votes
0answers
14 views

Pairwise vs aggregate A/B/n hypothesis testing

I'm currently running a hypothesis test on multiple variants (say $n$ variants) of a webpage, trying to optimize conversion, and am unsure if I should run pairwise hypothesis tests of each variant ...
1
vote
3answers
103 views

Eliminating variation in sample size for A/B tests

Firstly, I'll apologise for my limited understanding on this topic. Let's say I am running an A/B test with a pool of 20,000 prospects and I split them 50/50 into A and B. The problem is, the ...
1
vote
0answers
187 views

What statistical test is best to compare ratios within each groups

I am doing a a/b test. I am interested in finding difference in ratio of revenue after the campaign vs before for both the groups. Both groups are random split from same population subjected to two ...
1
vote
0answers
42 views

Possible questions on testing on hypothesis in A/B testing for a position of a data scientist (EDITED)

What are some possible machine learning and statistics questions on the subject of A/B testing for a position of a data scientist in Ad Tech industry working in retargeting? I'm interviewing for such ...
0
votes
1answer
19 views

MVT/ABn pricing test comparison

I have results for a a 5 variant pricing test: A 0% (control group) B +5% C -5% D +10% E -10% Below are my results (all normalised to give if exactly 20% of traffic goes through the variant, ...
2
votes
1answer
25 views

Which of these scenarios are multiple or repeated comparisons?

I believed I knew what multiple comparison or repeated testing meant but upon reading more on the subject and listening to others I have become more confused. My confusion started with A/B testing ...
1
vote
0answers
28 views

What would be a good sampler from pymc3 for highly skewed data

I have a gamma distributed data which is highly skewed - alpha=0.15, beta=0.001. I would like to perform mcmc to find the delta between two gamma distributions. I get the following error: I suspect ...
0
votes
1answer
32 views

P-value and confidence interval in MVT testing

Context: website A/B testing When calculating P-value and confidence intervals for each variation in an experiment with more than 2 variations (control, variation 1, variation 2 ...) should there be ...
1
vote
0answers
90 views

Sampling A/B test results (revenue per visitor vectors)

I have two vectors of the control version A and test version B. These vectors contain revenues by visitor. So A version has 3020 visitors who didn't purchase and B respectively 2811. Revenue data ...
5
votes
1answer
59 views

Do I need to discard 90% of experiments so that the sample is independent?

I'm reading OpenIntro Statistics by David Diez and he says that for running an inference on a one sample mean we need to validate that the observations are independent: I'm running an A/B test and I ...
0
votes
0answers
26 views

How to measure metric baseline?

During planning of an online A/B test, after we choose a success metric, we need to estimate the sample size required to detect a minimum effect. Usually this formula[1] is used: $n = 16 \frac{σ^2}{δ^...
0
votes
0answers
30 views

Weighted and Paired-comparison tests

We have an application with 7 different update versions. The goal is to prove to the customers that updating to the newer version is beneficial for you, since you will face with less bugs. The ...
1
vote
1answer
45 views

Non-inferiority margin and minimum detectable effect vs sample size

I am trying to understand why testing for non-inferiority requires pretty much the same sample size as testing for superiority (I assume the latter is the same as a one-sided test for a given MDE). I ...
1
vote
1answer
49 views

A/B testing parity (A=B)

I can't find any information on how to run an A/B test that shows that the new implementation B is likely at least as good as A. My metric is visitor conversion so I could use a test of proportions ...
1
vote
0answers
144 views

Understanding interim analysis and alpha spending functions

I have to perform an A/B test with interim analysis and found some literature about methods designed to preserve the global Type I error rate by adjusting the $\alpha$ level for each peeking. ...
1
vote
1answer
45 views

Confidence boundaries for an AB test [closed]

Given an AB-test, I have two variants running over a certain timeperiod gathering data. I split the time period up in smaller frames and cumulative add the population for the given variant as well as ...
0
votes
2answers
29 views

experiment with 3 arms: analyzing results when one of 3 pairwise tests shows significance

I have an experiment with 3 groups: group 1 got no email group 2 got an email with 10 dollars incentive group 3 got an email with 20 dollars incentive to convert to a specific product. Conversion ...
0
votes
1answer
50 views

How do I measure impact of an intervention of time series without historical data of the same series?

I'm trying to evaluate the impact of an intervention in the most statistically accurate way possible but I'm having a hard time figuring out a way measure incremental growth along with statistical ...
4
votes
0answers
3k views

Assessing A/B test results using Python

Problem's setting Assume having results of an A/B test. You let you users experience two variants of your website and you counted how many converted: ...
0
votes
1answer
237 views

Checking independence of A/B/C/… test results using Python

Assume I collected the following results from an A/B/C/D/E test: ...
1
vote
0answers
73 views

What is the practical difference between different significance calculators for A/B tests?

As background information I could say that we are currently studying A/B testing in University digital marketing class. As I have quite little background in statistics I am little bit wondering about ...
1
vote
1answer
560 views

Confidence Intervals for Mean Difference between Actuals and Forecast

I am trying to assess the influence of a product change compared to a hypothetical forecast. I use a time series model to predict a realistic future and compare it to the observed actuals. I am ...
0
votes
1answer
80 views

What role does standard deviation play in hypothesis testing?

I'm new to hypothesis testing in statistics. Say if we were to run an experimental trial on two control groups (split of 50% each), and after a large enough sample size, conclude by looking at just "...
0
votes
1answer
369 views

What is difference of Difference in Difference(DiD) and A/B Testing?

I am learning DiD and find it similar to A/B Testing. What's the difference between these two testing methods?