Questions tagged [ab-test]

A/B testing, also known as split or bucket testing, is a controlled comparison of the effectiveness of variants of a website, email, or other commercial product.

Filter by
Sorted by
Tagged with
2
votes
1answer
38 views

How to calculate statistical significance of an A/B test?

I am running an A/B/C test where users are shown three different introductions for a subscription service and they can then choose to receive weekly, monthly, or no content. I am tracking how many ...
0
votes
0answers
12 views

Multiple Goals with Bayesian A/B Testing

I am transitioning my team from frequentist to Bayesian A/B testing and have spent a lot of time researching all the possible questions and concerns that people may have. The one critical question ...
0
votes
1answer
66 views

Test if population sizes are statistically different?

I'm running an A/B test to see if the improved website layout increases revenue. First, I would like to check if the sizes of the control and experiment groups are statistically different. I have data ...
1
vote
0answers
48 views

Imbalanced classification or Regression? What is the best approach to my A/B testing related problem?

The context of the problem is A/B testing of two new versions of a game. I have a structured dataset (50000 rows x 22 columns) from the game designers that represents data with respect to two versions ...
1
vote
1answer
21 views

ANOVA complete block design, more units per block than treatments

I would like to design a randomized complete block design experiment (RCBD). Let's say I have 3 treatments and 10 logical groupings of my experimental units (EUs) which are the blocks. If each of my ...
0
votes
0answers
15 views

Which variant should be selected at the end of a multi-armed bandit(MAB) test?

The question is in the context of online experimentation, i.e. A/B testing. I understand we can use MAB testing to maximize conversions during a test window. But if we want to select a variant to be ...
3
votes
2answers
49 views

Email open-rate test

We ran an AB test for four days, sending our clients the same emails but with different headings, in order to know which header logic is better (i.e. has has the highest Open Rate (OR) = openings/ ...
0
votes
0answers
17 views

A/b testing mechanism - is there a problem or results skew is a coincidence

I have a system which does a/b testing on multiple web sites. I suspect that the a/b testing mechanism is skewed. I'm trying to figure out if the results indicate an issue in the mechanism or not. To ...
2
votes
1answer
47 views

Potential for bias to be introduced when limiting analysis to participants who seek treatment?

I am running an online AB test where users are assigned to condition (test or control) at login, but only 1/3 of assignees reach the feature being tested. Analyzing all assignees will give me an ...
0
votes
0answers
12 views

Removing Variance of Time Series?

I want to perform hypothesis testing on two "target" time series, a treatment and a control. I want to fit a GAM (Facebook's Prophet) and make a lower-variance estimate ("now-cast")...
0
votes
0answers
13 views

How to use Synth for comparative case studies?

I am trying to use synth for a comparative case study. I went through the documentation of the same, read the original research paper but I am still lost on how to actually implement it? which feature ...
3
votes
2answers
61 views

Optimal unbalanced design for A/B test

I want to find the right way to distribute variants (control/test) in A/B-test with the constraint of no more 10 percent of population in test group. The test is t-test for continuous metric. I can ...
0
votes
1answer
15 views

Calculating Duration of AB test for measuring sales change in stores

I have done my research about AB testing an seen that it is usually implemented in the online space(websites etc), I am trying to implement the same concept but at a store level. Problem: Given a ...
0
votes
0answers
16 views

Statistical comparison of response rates in two scored populations

I have the following dataset: Population 1 which was sent mailer-A : total size is 325818 responses are 74 the population was scored from 0 to 1 based on a regression model. A score of 1 means most ...
0
votes
0answers
13 views

What to conclude when the effect size is low in AB test and it takes very long to reach minimum sample size

We typically calculate the minimum sample size in our AB tests, based on the effect size we see between the test and control buckets (as minimum sample size is inverse to effect size, given power and ...
2
votes
1answer
46 views

How to detect sample size for multiple groups in an AB test?

I'm sure you know this sample size calculator by Evan Miller. Given necessary parameters, it outputs the efficient sizes for Control & Test groups. But what if I have multiple Test groups? What ...
1
vote
0answers
10 views

How to apply hypothesis or A/B test to 2 distributions obtained using 2 different methods

I am new to running statistical experiments. Thanks for your help in advance. Suppose now I have 2 different machine learning algorithm to enhance a picture's resolution from a neural networks. The ...
2
votes
1answer
31 views

Why is the suggested sample size much larger than what's necessary for significance?

Imagine you're doing an A/B-test in Marketing. You send a control group the standard e-mail, and a test group an e-mail in which one element was changed (for example the color of a button). You want ...
1
vote
1answer
33 views

How to compare conversion rates in an AB test?

Please help me decide how to proceed in the following situation: Suppose we have an AB test with 2 groups. The target metric is the conversion from one web page to another. Suppose also, that ...
0
votes
0answers
29 views

A/B testing use case

I am learning A/B testing and so far I have learned the theory and am trying to solve some use cases. I am working on a website data which has total visits and clicks at day x country x test/control ...
0
votes
0answers
38 views

Sample size in A/B testing and its impact on test being positive

How do I choose the right sample size for A/B test? looks like if I increase the sample size the test can always result in positive Do we use t-test if my sample size is more than 30? because in t-...
0
votes
0answers
17 views

Is CLT and AB test valid , when the test have repeat visitors making repeat pruchases

The Central limit theorem states that variables in population of the mean should be independent. But, when doing an AB test in a website, we have large numbers of visitors that will be coming back to ...
0
votes
0answers
11 views

Difficulty understanding the intuition behind two-population z-test

I'm trying to understand how I would compare conversion rate data in an A/B test. I've read a lot of posts on this, all of which generally boil down to computing a z-score using something like $$ z=\...
0
votes
0answers
35 views

How to Determine if Group A is Significantly Better than Group B

Edit: Please Find the Table in Image I am conducting research on 2 groups and lets call them A and B and these 2 groups are 2 Variables of my dataset. These are categorical data.. Below is sample of ...
0
votes
0answers
18 views

AB/Hypothesis testing in multiple groups

Let's say I have two variants of my website and I would like to know which variant generates more revenue. This website operates in multiple countries and I will choose a variant for all countries at ...
1
vote
1answer
73 views

A/B Testing in Time Series - i.i.d.?

If you're using the difference in differences method to run an A/B test on two time series (lets say the outcome measure is clickthrough rate), is there an i.i.d. assumption? More specifically, if ...
0
votes
0answers
38 views

A/B Testing: Reconciling sample size calculation with post test analysis

I'm trying to reconcile the sample size calculation pre-test vs post-test analysis. For pre-test, I am using the following link: https://www.evanmiller.org/ab-testing/sample-size.html. It suggests ...
0
votes
2answers
54 views

What statistical test to use for categorical data with 3 or more groups

We conducted A/B/C test in our website and now i want to check the results. But the problem i faced is that there are 2 test groups and 1 control, and not 1 test and 1 control groups as usual. When ...
0
votes
1answer
20 views

How to calculate the significance of the difference in average sales in an A/B test

I am doing A/B testing for ad display, where the wording on the ad is different between the two groups. Metrics gathered: 1) # of people the ad reached 2) # of people who clicked on the ad and were ...
0
votes
1answer
26 views

Can you combine the results of two A/B tests?

Suppose, a feature of an app is redesigned and A/B tested in Region A with a small population, where x% of this population found the feature to be good, and it improved user engagement on the app. ...
0
votes
1answer
53 views

A/B testing and Multi-armed bandit algorithms in a recommender system

I was reading about these two algorithms, but I don't understand how they can be used in a recommender system because using the MovieLens dataset these algorithms recommend the best movie for all the ...
0
votes
0answers
28 views

AB Test Cross Contamination

I was talking with consultancy today about running geo experiments. The idea is to randomise on a zip code level instead of a user level for exposing customers to ads online. The issue is, there will ...
1
vote
1answer
63 views

Are p-value and Bayesian probabilities equivalent when applied to conversion rates?

For evaluating AB-tests, one can use the frequentist approach or a Bayesian one. Using these websites I compared both with some example data: https://abtestguide.com/bayesian/ https://abtestguide....
0
votes
0answers
16 views

Does ML Continuous Learning Require AB testing?

ML continuous learning is the process of updating your ML in production with new data. Does the updated model need to be AB tested against the original? What's the process for this?
0
votes
0answers
12 views

Summary stats for studies with correlated observations

I have two experiments I ran recently (simple random sampling), where there is within user correlation. To be more clear, experiment 1 is looking at clickthrough rates between a control and an ...
0
votes
0answers
33 views

t-test for means of daily conversion

Let's say I'm a/b testing a new site version and want to know if it improves conversion. I realize that conversion is a proportion so I would use a Z-test for this. But then I am thinking - it is ...
0
votes
0answers
69 views

Accounting for Pre-Period Bias in AB Testing with Logistic Regression

Question Context This is more of a conceptual question although a concrete example is provided for clarity. My company uses AB testing to understand the impact of various email campaigns on customer ...
0
votes
0answers
12 views

Experimentation Time series - Find a control group whose mean follows closely the test group

I'm conducting an analysis from Jan to Dec where my test/treatment group is the combination of 5 stores with sales per month. The main objective is to find a control group of 10 stores over 150 ...
0
votes
0answers
57 views

the role of MDE in a/b test analysis

I have a question about the role of the minimum-detectable-difference (MDE) in power analysis before the experiment is run and in post-experiment analysis. To clarify: suppose I am testing the ...
0
votes
0answers
35 views

Optimizing profitability by adjusting payment terms

I'm trying to optimize profitability by adjusting payment terms. My company sells mostly to small businesses; some of these customers receive payment terms (e.g., net 30 days) and some are required ...
1
vote
1answer
427 views

P-value calculation

I am trying to build P-value calculation within Tableau for A/B tests and found a formula. But the formula doesn't work all the time. ...
0
votes
0answers
46 views

Sample size calculation in AB testing

In AB testing, how to calculate the minimum sample size? is there one standard way or there are multiple ways to do the calculation? Does the population size matter?
3
votes
1answer
64 views

Hypothesis test of two population means with correlated observations

I'd like to run an A/B experiment where my outcome measures are conversion rate, and mean spend. I'd like to use two proportion Z-test and z-test of two population means, however many users in the ...
1
vote
1answer
108 views

Calculate the difference in sums (revenue) between two groups in A/B test

Requests for a quote come in completely independently, and our system assigns each to either A or B randomly (50/50% split). B has slightly different logic, that is what we are testing. Quotes ...
14
votes
3answers
1k views

When is NHST appropriate in business?

Null hypothesis significance testing seems to be widely used in business. The most obvious example is A/B Testing, where a business will perform a test comparing two variants of some aspect of their ...
1
vote
1answer
48 views

Bayesian A/B Testing vs Frequentist A/B Testing?

I was reading the book Probabilistic Programming for Hackers and came across Bayesian A/B Testing. So basically, I get a Posterior Distribution for $P(A), P(B)$ and the $\Delta$ between $P(A)$ and $P(...
0
votes
2answers
40 views

A/B Test - P Value is 1

I am running an A/B test for our site. And get below stats: Group A: Visitors: 155,590; Conversion: 18,529 Group B: Visitors: 155,592; Conversion: 18,118 And I am using the tool on this website to ...
0
votes
0answers
33 views

Question on this study design

If I'm looking to run a randomized controlled experiment comparing: Group A) control Group B) treatment 1 Group C) treatment 2 Group D) a combination of treatments 1 and 2 1) Can I reduce ...
2
votes
1answer
103 views

Statistical Significance of A/B Test for Binary outcome

My situation is I have a list of people that were included in a marketing campaign. Half the list got version 1 of an email and the other half got version 2. Then people either bought the product or ...
1
vote
1answer
64 views

A/B Testing in observational studies

In the regime of observational studies, I am trying to understand if system A works better then system B. The task of a system is to pick products from a given cart full of products. Whether a pick ...

1
2 3 4 5