Questions tagged [ab-test]

A/B testing, also known as split or bucket testing, is a controlled comparison of the effectiveness of variants of a website, email, or other commercial product.

198 questions
Filter by
Sorted by
Tagged with
14 views

How does Thomson Sampling work in a real world application of Multi Armed Bandit Testing

I understand the basics of Thomson Sampling, but how is it implemented in practice? If there are three variants each with a 1/3 of traffic allocated to them on day 1, how is traffic dynamically ...
71 views

Analyzing ABn test when normality is not met

I'm analyzing an unpaired A/B/C/D test to assess whether the mean or median of population B/C/D is higher than the one of A. As shown from the left plot the normality distribution cannot be assumed. ...
33 views

Determining the significance of rare events

We have been having a discussion about the significance of 2 accidents. If I have 2 groups: Group 1 is a baseline where an accident has occurred 70 times in 3,500,000 and Group 2 where an accident ...
30 views

Evaluating different methodologies for hypothesis testing

I have a campaign, with a target and holdout group, in which the target group is encouraged to make a purchase. I have the results from campaign - total revenue spent per member (for target & ...
20 views

Can you use power as a probability to calculate expected outcome in A/B tests?

Background: I'm trying to think about a cost/benefit framework for decision making based on results from an A/B test, my question is specifically about the use of power as a probability in a ...
52 views

Basic AB Testing with R

Suppose there is an email campaign for customers. There are two equally divided groups A and B. B received an email with "Offer ends this Saturday. Use code B!". A did not receive the email. How ...
32 views

A/B Test Statistical Analysis - Margin of Error

We ran a number of tests (A/B 50/50 split) and replicated over five days, with five rows of results: (EXAMPLE DATA STRUCTURE) ...
59 views

How to interpret bayesian posterior distributions

I think that I miss something very basic about the posterior distributions in bayesian hypothesis testing. In the frequentist approach, we define the null hypothesis, compute the test power, define ...
32 views

Test of ratio within population

Let's say I have two large containers, X and Y, both containing a mix of red and blue buttons (for clothes). Consider the unknown ratios $x$ := #blue / (#blue + #red) in container X $y$ := #blue / (#...
18 views

How do I calculate sample sizes for multiple proportions, while maintaining a certain ratio of the sample sizes?

I am trying to perform z-tests for multiple proportions, while maintaining a certain ratio of the samples. For example, let's say I'm trying to compare 3 populations, and I want to maintain the ...
47 views

Is it valid to use t-test when using ratio of two variants

Consider a setup where it is not possible to show different variants to different users. But we still would like to measure the performance of different variants. We would like to use the following ...
106 views

How to select the best variant from an A/B/C test?

Suppose I've run an A/B/C experiment (same as A/B but with 3 groups instead of 2) and gathered the following data for number of participants in each group and number of desired actions in each group (...
68 views

How would I run an A/B test if the observations are very right­-skewed?

As the title states, how would I run an A/B test if the observations are very right­-skewed? What could I do in order to still have a valid result? Should I remove outliers?
31 views

Comparisons of targeting models

I need to compare two targeting models in an experiment. We will apply the same treatment to everyone who is targeted. The experimental set up is as follows: The total population is split into ...
70 views

Sample size prediction in multivariate A-B testing

I need to run an A-B testing with one control group (40%) and 3 variants (20%) each. Base line = 0.65 Target lift = 0.25 Power =0.8 Confidence = 95% I found following R code to find minimum ...
53 views

What is the equation to calculate the sample size for A/B testing? [closed]

I have been searching for information on how to calculate sample size for A/B testing. My research was fruitless. I found many online calculators, but not any blog/paper that explains how the ...
24 views

Why we have to use pooled sample proportion to calculate the pooled standard deviation of the difference?

The context can be found in Hypothesis Test: Difference Between Proportions It says there: ...
22 views

Need help determining sample size for an AB test with two metrics

I need to run an experiment with customer support agents who are rated on two metrics by the customer: average issue resolution rate (proportion of issues resolved) and average star rating (1-5). ...
30 views

Bayesian A/B testing and error estimates

Someone from my work started using an AB calculator that they found online and asked my take on the approach which i assume is incorrect. The approach takes the conversion and visits for control and ...
43 views

Measuring incremental revenue / uplift

I run a campaign with an experimental design setup. The target and control group is representative and split 70/30. The target group receives an email communication The dynamics of the campaign ...
44 views

What is necessary sample size for B in A/B test with unequal sample sizes?

I have a situation in which I want to measure if the error rate in system B (test) is worse than the error rate in system A (control). System A has a sample size of 100 and that cannot be modified. So ...
8 views

Is there a way to run test for A vs B for metric cost per lead = marketing cost / lead?

Is there a way to run test for A vs B for metric cost per lead = marketing cost / lead? the denominator is the total number of leads submitted from the marketing audience. the numerator is the ...
618 views

2 sample t test for sample sizes - 30,000 and 150,000

I have 2 samples, one with sample size of 30,000 customers and the other with 150,000. I have to perform a 2 sample t test(on conversion rates of the 2 groups). My question is, will t test in this ...
27 views

Interpretation for Chi-Square Test for Independence in A/B/n testing

I am testing the results of a marketing campaign which involves randomly splitting a group of people into 4 treatments. One group acts as a control and is shown no advertising, the other 3 groups are ...
58 views

AB Test for Total Revenue: How to take both of conversion rate and average order value into account in my ab-test?

I have some purchase data for two groups of people, for instance: Method Revenue ------ ------- A 12 A 0 A 0 A 4 A 0 B 0 B 8 B ...
46 views

AB testing and hypothesis testing

I have an entire market for which I plan to run two offers. I randomly split the market 50/50. All of Group A receives offer 1 and all of group B receives offer 2. I want to compare proportions of ...
102 views

A/B testing for binary classification tasks

I work on project where each user has an IoT sensor and an ML algorithm tries to identify a given situation. The user carries the sensor 24/7 but I split the time into 'sessions'. For each session, I ...
26 views

Confidence interval in Chi squared testing

How to calculate confidence interval when performing a Chi-Square significance testing? For reference where it's used - http://www.evanmiller.org/ab-testing/chi-squared.html Can we apply the same ...
751 views

A/B testing ratio of sums

Context Consider the following scenario for a company selling goods online. A user can purchase several items (i.e. basket of items), some of which are of particular importance and are tracked ...
81 views

How to calculate significance for an A/B test for average revenue per daily active user over a 1-week test period?

One simple approach is to use "per active day" metrics. e.g. "revenue_per_active_day" . However there is a fundamental flaw with this approach. e.g. during a 7-day period, let's say user 1 is active ...
202 views

AB Testing vs Hypothesis Testing

I'm having trouble making the full connection between AB Testing and Hypothesis testing. Hypothesis Testing Conducting a test to compare two means, we need the standard deviation of both samples, and ...
60 views

statistical significance of A B test if some observation are removed due to errors

I have ran an A/B test and using chi-square test of independence, results shows that group A improves on group B by less than 5% (statistically significant) YAY!!! Unfortunately, after few checks I ...
864 views

AB test sample size calculation by hand

Evan Miller has created a well-known online AB test sample size calculator. For the sake of being able to program and modify this formula, I would like to know how to calculate sample size Evan Miller-...
26 views

A\B Testing with Sorting

In my company while splitting site users for AB test we do the following: we choose parameters that seem the most important for us (for example: gender, life time value, average check, etc), if they ...
24 views

how to present results for two group success rate with confidence interval

I have two groups A and B, and for each data point I have two possible outcomes, success and failures. For each group, I have calculated the success rate, and the confidence interval for a binomial ...
4 views

Statistics of Diff in diff with optimal group

I am trying to run a diff in diff to measure the impact of a feature. I want to run the experiment per geo region and I have 90 regions. To run a proper diff in diff I should split in have the ...
15 views

Conversion/Revenue Attribution in AB experiments

I have a question on A/B testing attribution. I have a test running on mobile app and website at the same time. Unfortunately the way the test is designed ...
21 views

In AB test do the sample size calculations represent the minimum or maximum size required for measuring difference between the test and control

A quick question on the frequentest approach of sample size calculation. I recently came across this below note: The sample size required should be strictly enforced; if the power analysis shows ...
21 views

A/B test : Is there a way interpreting factor “A” group is greater than factor “B” group?

Plz.. understand my lack of English skills.. T.T I have a problem dealing with statistical hypothesis testing. To be specific, we have two special delivery method like (new) : method "A", (old) : ...
35 views

Good numbers, but something feels wrong

This is more a qualitative question than a quantitive one. I have done an A B test to try to increase use of a particular feature (feature 1) by increasing discoverability. I measure it by seeing ...
484 views

Multi-armed bandit vs AB testing

I'm trying to understand the difference between AB Testing and Multi armed bandit and when we should use one of the other. It seems to me that Multi-armed bandit algorithms are statistically valid and ...
168 views

Eliminating variation in sample size for A/B tests

Firstly, I'll apologise for my limited understanding on this topic. Let's say I am running an A/B test with a pool of 20,000 prospects and I split them 50/50 into A and B. The problem is, the ...
353 views

What statistical test is best to compare ratios within each groups

I am doing a a/b test. I am interested in finding difference in ratio of revenue after the campaign vs before for both the groups. Both groups are random split from same population subjected to two ...
46 views

Possible questions on testing on hypothesis in A/B testing for a position of a data scientist (EDITED)

What are some possible machine learning and statistics questions on the subject of A/B testing for a position of a data scientist in Ad Tech industry working in retargeting? I'm interviewing for such ...
30 views

MVT/ABn pricing test comparison

I have results for a a 5 variant pricing test: A 0% (control group) B +5% C -5% D +10% E -10% Below are my results (all normalised to give if exactly 20% of traffic goes through the variant, ...
25 views

Which of these scenarios are multiple or repeated comparisons?

I believed I knew what multiple comparison or repeated testing meant but upon reading more on the subject and listening to others I have become more confused. My confusion started with A/B testing ...
43 views

What would be a good sampler from pymc3 for highly skewed data

I have a gamma distributed data which is highly skewed - alpha=0.15, beta=0.001. I would like to perform mcmc to find the delta between two gamma distributions. I get the following error: I suspect ...
38 views

P-value and confidence interval in MVT testing

Context: website A/B testing When calculating P-value and confidence intervals for each variation in an experiment with more than 2 variations (control, variation 1, variation 2 ...) should there be ...