Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [ab-test]

A/B testing, also known as split or bucket testing, is a controlled comparison of the effectiveness of variants of a website, email, or other commercial product.

0
votes
1answer
17 views

In AB test do the sample size calculations represent the minimum or maximum size required for measuring difference between the test and control

A quick question on the frequentest approach of sample size calculation. I recently came across this below note: The sample size required should be strictly enforced; if the power analysis shows ...
0
votes
0answers
19 views

A/B test : Is there a way interpreting factor “A” group is greater than factor “B” group?

Plz.. understand my lack of English skills.. T.T I have a problem dealing with statistical hypothesis testing. To be specific, we have two special delivery method like (new) : method "A", (old) : ...
1
vote
0answers
34 views

Good numbers, but something feels wrong

This is more a qualitative question than a quantitive one. I have done an A B test to try to increase use of a particular feature (feature 1) by increasing discoverability. I measure it by seeing ...
0
votes
2answers
28 views

Multi-armed bandit vs AB testing

I'm trying to understand the difference between AB Testing and Multi armed bandit and when we should use one of the other. It seems to me that Multi-armed bandit algorithms are statistically valid and ...
0
votes
0answers
8 views

Pairwise vs aggregate A/B/n hypothesis testing

I'm currently running a hypothesis test on multiple variants (say $n$ variants) of a webpage, trying to optimize conversion, and am unsure if I should run pairwise hypothesis tests of each variant ...
1
vote
3answers
43 views

Eliminating variation in sample size for A/B tests

Firstly, I'll apologise for my limited understanding on this topic. Let's say I am running an A/B test with a pool of 20,000 prospects and I split them 50/50 into A and B. The problem is, the ...
0
votes
0answers
13 views

Testing that's there's no difference in conversion testing

I am looking at implementing a new website A/B testing tool. Before I turn everything over to it, I want to run a test where I divert traffic to two experiences that are exactly the same and show they ...
1
vote
0answers
68 views

What statistical test is best to compare ratios within each groups

I am doing a a/b test. I am interested in finding difference in ratio of revenue after the campaign vs before for both the groups. Both groups are random split from same population subjected to two ...
0
votes
0answers
12 views

How to compare a binary choice test?

We have two separate algorithms, A and B, which, given some input, produce some result. Users are to be shown multiple examples where each algorithm was given the same input, and for each input, ...
1
vote
0answers
41 views

Possible questions on testing on hypothesis in A/B testing for a position of a data scientist (EDITED)

What are some possible machine learning and statistics questions on the subject of A/B testing for a position of a data scientist in Ad Tech industry working in retargeting? I'm interviewing for such ...
0
votes
1answer
13 views

MVT/ABn pricing test comparison

I have results for a a 5 variant pricing test: A 0% (control group) B +5% C -5% D +10% E -10% Below are my results (all normalised to give if exactly 20% of traffic goes through the variant, ...
2
votes
1answer
23 views

Which of these scenarios are multiple or repeated comparisons?

I believed I knew what multiple comparison or repeated testing meant but upon reading more on the subject and listening to others I have become more confused. My confusion started with A/B testing ...
1
vote
0answers
17 views

What would be a good sampler from pymc3 for highly skewed data

I have a gamma distributed data which is highly skewed - alpha=0.15, beta=0.001. I would like to perform mcmc to find the delta between two gamma distributions. I get the following error: I suspect ...
0
votes
1answer
27 views

P-value and confidence interval in MVT testing

Context: website A/B testing When calculating P-value and confidence intervals for each variation in an experiment with more than 2 variations (control, variation 1, variation 2 ...) should there be ...
0
votes
0answers
34 views

How to implement bayesian ab testing to gamma distributed data using pymc3

I found a simple implementation for Bernoulli distributed data of the bayesian ab tesing using pymc3 in the Probabilistic Programming and Bayesian Methods for Hackers book: http://nbviewer.jupyter.org/...
0
votes
0answers
24 views

AB testing for low traffic sites

I am working on a AB test on a specific page that has very low traffic (like ~1200 per day). As most of us know, features with smaller traffic need more time to detect any significant change unless ...
1
vote
0answers
52 views

Sampling A/B test results (revenue per visitor vectors)

I have two vectors of the control version A and test version B. These vectors contain revenues by visitor. So A version has 3020 visitors who didn't purchase and B respectively 2811. Revenue data ...
0
votes
0answers
34 views

Statistical analysis to find causality?

I have the following two groups of data: (a) jobs done before a certain feature was released in our software (b) jobs done after the feature was released in our software The purpose of the feature ...
5
votes
1answer
58 views

Do I need to discard 90% of experiments so that the sample is independent?

I'm reading OpenIntro Statistics by David Diez and he says that for running an inference on a one sample mean we need to validate that the observations are independent: I'm running an A/B test and I ...
0
votes
0answers
24 views

How to measure metric baseline?

During planning of an online A/B test, after we choose a success metric, we need to estimate the sample size required to detect a minimum effect. Usually this formula[1] is used: $n = 16 \frac{σ^2}{δ^...
0
votes
0answers
20 views

Weighted and Paired-comparison tests

We have an application with 7 different update versions. The goal is to prove to the customers that updating to the newer version is beneficial for you, since you will face with less bugs. The ...
0
votes
0answers
14 views

Non-inferiority experiment: results fluctuate daily, significant on some days and not others

I am trying to interpret the results of an experiment after the fact (was not involved in setting up, determining sample size, etc). The experiment had the aim of 'not breaking anything', so there was ...
0
votes
0answers
10 views

Check for difference for 2 groups with 4 possible test results

I need to calculate sample size for test with 2 groups of people where test results are 4 possible outcomes: 1 - no, 2 - possibly no, 3 - possibly yes, 4 - yes. What kind of statistics test can I use ...
1
vote
1answer
26 views

Non-inferiority margin and minimum detectable effect vs sample size

I am trying to understand why testing for non-inferiority requires pretty much the same sample size as testing for superiority (I assume the latter is the same as a one-sided test for a given MDE). I ...
1
vote
1answer
46 views

A/B testing parity (A=B)

I can't find any information on how to run an A/B test that shows that the new implementation B is likely at least as good as A. My metric is visitor conversion so I could use a test of proportions ...
1
vote
0answers
97 views

Understanding interim analysis and alpha spending functions

I have to perform an A/B test with interim analysis and found some literature about methods designed to preserve the global Type I error rate by adjusting the $\alpha$ level for each peeking. ...
0
votes
0answers
21 views

Single Sample Hypothesis Z-test V.S. Two Populations, t-test with Hypothesis

for website A/B test, I split the traffic, 50% to control group, 50% to Test group. ...
1
vote
1answer
44 views

Confidence boundaries for an AB test [closed]

Given an AB-test, I have two variants running over a certain timeperiod gathering data. I split the time period up in smaller frames and cumulative add the population for the given variant as well as ...
0
votes
2answers
25 views

experiment with 3 arms: analyzing results when one of 3 pairwise tests shows significance

I have an experiment with 3 groups: group 1 got no email group 2 got an email with 10 dollars incentive group 3 got an email with 20 dollars incentive to convert to a specific product. Conversion ...
0
votes
1answer
42 views

How do I measure impact of an intervention of time series without historical data of the same series?

I'm trying to evaluate the impact of an intervention in the most statistically accurate way possible but I'm having a hard time figuring out a way measure incremental growth along with statistical ...
0
votes
0answers
129 views

Determining sample size for revenue and order item ab test

Suppose I have the following structure for a product sales funnel. A customer purchases product "A". Then they are presented with the option to purchase an "upsell" (upsell 1) with one click, which ...
4
votes
0answers
2k views

Assessing A/B test results using Python

Problem's setting Assume having results of an A/B test. You let you users experience two variants of your website and you counted how many converted: ...
0
votes
0answers
28 views

Impact evaluation without experimental design

Let's say you are a professor, and made a session on "How to prepare before a lecture ?". This session was optional, and only a fraction of students came. Still, you want to assess the impact your ...
0
votes
1answer
186 views

Checking independence of A/B/C/… test results using Python

Assume I collected the following results from an A/B/C/D/E test: ...
0
votes
0answers
26 views

A/B Testing - Appropriate Statistical Test?

We are testing the impact of website variations (A vs. B) on probability of users subscribing. Users consist of: new first-time visitors who 1st visit during the testing period prior / repeat ...
1
vote
0answers
62 views

What is the practical difference between different significance calculators for A/B tests?

As background information I could say that we are currently studying A/B testing in University digital marketing class. As I have quite little background in statistics I am little bit wondering about ...
0
votes
0answers
19 views

Need of experiment duration in an A/B test

The Law of Large Numbers states that as a sample size increases, the sample mean will get closer to the population mean. But when we run a fixed sample A/B test, why don't we consider it running for a ...
1
vote
1answer
418 views

Confidence Intervals for Mean Difference between Actuals and Forecast

I am trying to assess the influence of a product change compared to a hypothetical forecast. I use a time series model to predict a realistic future and compare it to the observed actuals. I am ...
0
votes
1answer
51 views

What role does standard deviation play in hypothesis testing?

I'm new to hypothesis testing in statistics. Say if we were to run an experimental trial on two control groups (split of 50% each), and after a large enough sample size, conclude by looking at just "...
0
votes
1answer
228 views

What is difference of Difference in Difference(DiD) and A/B Testing?

I am learning DiD and find it similar to A/B Testing. What's the difference between these two testing methods?
-1
votes
2answers
900 views

What does statistically insignificant mean?

Say if an A/B test proves to be insignificant but the test group shows that its doing better than the control group, what would be some of the reasons that the test is insignificant?
1
vote
0answers
27 views

A/B testing on sparse metrics

Out of my entire population(in this case all the possible posts) only .05% of my posts will be liked. I introduce a new post format, and design an A/B test to test my old post format to see if my new ...
0
votes
1answer
93 views

Selecting Bayesian priors for the dependent data

I have goal of measuring CTRs of several titles of an article on a website using Bayesian approach. In a simple setup, what one will do is to select Beta Prior (for example Uniform distribution) and ...
0
votes
1answer
30 views

Contaminated AB testing

Suppose that you are a product manager who wants to run a test for a new product. You consult your statistician, select a sample of customers and run the test. Apparently, another product manager ...
0
votes
0answers
17 views

Why creating groups for A/B split tests

I wonder, why I have to assign users to groups for A/B tests, say a layout split for an ad on a webpage. Why is it not simply possible to show each user a different ad layout for each page impression? ...
0
votes
1answer
64 views

Are A/B testing problems intrinsically a multiple testing problem?

Can we convert any A/B testing problem into a multiple testing problem?
0
votes
1answer
86 views

Multiple A/B test of proportion in R

I'm currently facing the following situation. We have run a marketing campaign providing to some members one of two type of coupones. In addition to this, some of these members were already contacted ...
1
vote
0answers
25 views

Confirming AB test result intuition

Say I am running an AB test (control = $A$ and test = $B$). My significance level is 5% and power is 80%. My null hypothesis is that $p_A = p_B$ vs $p_A \ne p_B$. My historical conversion rate for ...
2
votes
1answer
279 views

Complexities of interpreting A/B testing power & significance calculations

Apologies up front for not being a statistician. I've been searching high and low and I cannot find a sufficiently detailed explanation how to interpret differences in A/B testing power and ...