Questions tagged [ab-test]

A/B testing, also known as split or bucket testing, is a controlled comparison of the effectiveness of variants of a website, email, or other commercial product.

Filter by
Sorted by
Tagged with
0
votes
1answer
26 views

A/B testing and Multi-armed bandit algorithms in a recommender system

I was reading about these two algorithms, but I don't understand how they can be used in a recommender system because using the MovieLens dataset these algorithms recommend the best movie for all the ...
0
votes
0answers
17 views

AB Test Cross Contamination

I was talking with consultancy today about running geo experiments. The idea is to randomise on a zip code level instead of a user level for exposing customers to ads online. The issue is, there will ...
0
votes
0answers
14 views

How to compute MSE in A/B testing?

How can I compute MSE in A/B testing? I have the A/B testing curve and the right value of the prediction but I mean I just have the optimum value
1
vote
1answer
60 views

Are p-value and Bayesian probabilities equivalent when applied to conversion rates?

For evaluating AB-tests, one can use the frequentist approach or a Bayesian one. Using these websites I compared both with some example data: https://abtestguide.com/bayesian/ https://abtestguide....
0
votes
0answers
13 views

Does ML Continuous Learning Require AB testing?

ML continuous learning is the process of updating your ML in production with new data. Does the updated model need to be AB tested against the original? What's the process for this?
0
votes
0answers
11 views

Summary stats for studies with correlated observations

I have two experiments I ran recently (simple random sampling), where there is within user correlation. To be more clear, experiment 1 is looking at clickthrough rates between a control and an ...
0
votes
0answers
21 views

t-test for means of daily conversion

Let's say I'm a/b testing a new site version and want to know if it improves conversion. I realize that conversion is a proportion so I would use a Z-test for this. But then I am thinking - it is ...
0
votes
0answers
19 views

Accounting for Pre-Period Bias in AB Testing with Logistic Regression

Question Context This is more of a conceptual question although a concrete example is provided for clarity. My company uses AB testing to understand the impact of various email campaigns on customer ...
0
votes
0answers
8 views

Experimentation Time series - Find a control group whose mean follows closely the test group

I'm conducting an analysis from Jan to Dec where my test/treatment group is the combination of 5 stores with sales per month. The main objective is to find a control group of 10 stores over 150 ...
0
votes
0answers
17 views

the role of MDE in a/b test analysis

I have a question about the role of the minimum-detectable-difference (MDE) in power analysis before the experiment is run and in post-experiment analysis. To clarify: suppose I am testing the ...
0
votes
0answers
32 views

Optimizing profitability by adjusting payment terms

I'm trying to optimize profitability by adjusting payment terms. My company sells mostly to small businesses; some of these customers receive payment terms (e.g., net 30 days) and some are required ...
1
vote
1answer
96 views

P-value calculation

I am trying to build P-value calculation within Tableau for A/B tests and found a formula. But the formula doesn't work all the time. ...
0
votes
0answers
40 views

Sample size calculation in AB testing

In AB testing, how to calculate the minimum sample size? is there one standard way or there are multiple ways to do the calculation? Does the population size matter?
3
votes
1answer
41 views

Hypothesis test of two population means with correlated observations

I'd like to run an A/B experiment where my outcome measures are conversion rate, and mean spend. I'd like to use two proportion Z-test and z-test of two population means, however many users in the ...
1
vote
1answer
43 views

Calculate the difference in sums (revenue) between two groups in A/B test

Requests for a quote come in completely independently, and our system assigns each to either A or B randomly (50/50% split). B has slightly different logic, that is what we are testing. Quotes ...
14
votes
3answers
1k views

When is NHST appropriate in business?

Null hypothesis significance testing seems to be widely used in business. The most obvious example is A/B Testing, where a business will perform a test comparing two variants of some aspect of their ...
1
vote
1answer
33 views

Bayesian A/B Testing vs Frequentist A/B Testing?

I was reading the book Probabilistic Programming for Hackers and came across Bayesian A/B Testing. So basically, I get a Posterior Distribution for $P(A), P(B)$ and the $\Delta$ between $P(A)$ and $P(...
0
votes
2answers
38 views

A/B Test - P Value is 1

I am running an A/B test for our site. And get below stats: Group A: Visitors: 155,590; Conversion: 18,529 Group B: Visitors: 155,592; Conversion: 18,118 And I am using the tool on this website to ...
0
votes
0answers
31 views

Question on this study design

If I'm looking to run a randomized controlled experiment comparing: Group A) control Group B) treatment 1 Group C) treatment 2 Group D) a combination of treatments 1 and 2 1) Can I reduce ...
2
votes
1answer
51 views

Statistical Significance of A/B Test for Binary outcome

My situation is I have a list of people that were included in a marketing campaign. Half the list got version 1 of an email and the other half got version 2. Then people either bought the product or ...
1
vote
1answer
42 views

A/B Testing in observational studies

In the regime of observational studies, I am trying to understand if system A works better then system B. The task of a system is to pick products from a given cart full of products. Whether a pick ...
1
vote
1answer
39 views

Meaning of Power Analysis function parameters for SSizeLogisticBin when A/B test in R

I did recently an A/B testing course in R from DataCamp and I could not understand the parameter B from the following function from the package powerMediation ...
0
votes
1answer
40 views

How does Thomson Sampling work in a real world application of Multi Armed Bandit Testing

I understand the basics of Thomson Sampling, but how is it implemented in practice? If there are three variants each with a 1/3 of traffic allocated to them on day 1, how is traffic dynamically ...
1
vote
2answers
75 views

Analyzing ABn test when normality is not met

I'm analyzing an unpaired A/B/C/D test to assess whether the mean or median of population B/C/D is higher than the one of A. As shown from the left plot the normality distribution cannot be assumed. ...
1
vote
1answer
95 views

Determining the significance of rare events

We have been having a discussion about the significance of 2 accidents. If I have 2 groups: Group 1 is a baseline where an accident has occurred 70 times in 3,500,000 and Group 2 where an accident ...
0
votes
0answers
31 views

Evaluating different methodologies for hypothesis testing

I have a campaign, with a target and holdout group, in which the target group is encouraged to make a purchase. I have the results from campaign - total revenue spent per member (for target & ...
1
vote
0answers
21 views

Can you use power as a probability to calculate expected outcome in A/B tests?

Background: I'm trying to think about a cost/benefit framework for decision making based on results from an A/B test, my question is specifically about the use of power as a probability in a ...
0
votes
1answer
61 views

Basic AB Testing with R

Suppose there is an email campaign for customers. There are two equally divided groups A and B. B received an email with "Offer ends this Saturday. Use code B!". A did not receive the email. How ...
0
votes
1answer
35 views

A/B Test Statistical Analysis - Margin of Error

We ran a number of tests (A/B 50/50 split) and replicated over five days, with five rows of results: (EXAMPLE DATA STRUCTURE) ...
5
votes
1answer
80 views

How to interpret bayesian posterior distributions

I think that I miss something very basic about the posterior distributions in bayesian hypothesis testing. In the frequentist approach, we define the null hypothesis, compute the test power, define ...
1
vote
1answer
34 views

Test of ratio within population

Let's say I have two large containers, X and Y, both containing a mix of red and blue buttons (for clothes). Consider the unknown ratios $x$ := #blue / (#blue + #red) in container X $y$ := #blue / (#...
0
votes
0answers
21 views

How do I calculate sample sizes for multiple proportions, while maintaining a certain ratio of the sample sizes?

I am trying to perform z-tests for multiple proportions, while maintaining a certain ratio of the samples. For example, let's say I'm trying to compare 3 populations, and I want to maintain the ...
1
vote
1answer
58 views

Is it valid to use t-test when using ratio of two variants

Consider a setup where it is not possible to show different variants to different users. But we still would like to measure the performance of different variants. We would like to use the following ...
2
votes
2answers
118 views

How to select the best variant from an A/B/C test?

Suppose I've run an A/B/C experiment (same as A/B but with 3 groups instead of 2) and gathered the following data for number of participants in each group and number of desired actions in each group (...
0
votes
1answer
80 views

How would I run an A/B test if the observations are very right­-skewed?

As the title states, how would I run an A/B test if the observations are very right­-skewed? What could I do in order to still have a valid result? Should I remove outliers?
0
votes
0answers
31 views

Comparisons of targeting models

I need to compare two targeting models in an experiment. We will apply the same treatment to everyone who is targeted. The experimental set up is as follows: The total population is split into ...
1
vote
1answer
170 views

Sample size prediction in multivariate A-B testing

I need to run an A-B testing with one control group (40%) and 3 variants (20%) each. Base line = 0.65 Target lift = 0.25 Power =0.8 Confidence = 95% I found following R code to find minimum ...
0
votes
1answer
70 views

What is the equation to calculate the sample size for A/B testing? [closed]

I have been searching for information on how to calculate sample size for A/B testing. My research was fruitless. I found many online calculators, but not any blog/paper that explains how the ...
1
vote
0answers
80 views

Why we have to use pooled sample proportion to calculate the pooled standard deviation of the difference?

The context can be found in Hypothesis Test: Difference Between Proportions It says there: ...
1
vote
1answer
28 views

Need help determining sample size for an AB test with two metrics

I need to run an experiment with customer support agents who are rated on two metrics by the customer: average issue resolution rate (proportion of issues resolved) and average star rating (1-5). ...
1
vote
0answers
32 views

Bayesian A/B testing and error estimates

Someone from my work started using an AB calculator that they found online and asked my take on the approach which i assume is incorrect. The approach takes the conversion and visits for control and ...
1
vote
1answer
76 views

Measuring incremental revenue / uplift

I run a campaign with an experimental design setup. The target and control group is representative and split 70/30. The target group receives an email communication The dynamics of the campaign ...
1
vote
1answer
73 views

What is necessary sample size for B in A/B test with unequal sample sizes?

I have a situation in which I want to measure if the error rate in system B (test) is worse than the error rate in system A (control). System A has a sample size of 100 and that cannot be modified. So ...
0
votes
0answers
8 views

Is there a way to run test for A vs B for metric cost per lead = marketing cost / lead?

Is there a way to run test for A vs B for metric cost per lead = marketing cost / lead? the denominator is the total number of leads submitted from the marketing audience. the numerator is the ...
3
votes
3answers
678 views

2 sample t test for sample sizes - 30,000 and 150,000

I have 2 samples, one with sample size of 30,000 customers and the other with 150,000. I have to perform a 2 sample t test(on conversion rates of the 2 groups). My question is, will t test in this ...
0
votes
0answers
35 views

Interpretation for Chi-Square Test for Independence in A/B/n testing

I am testing the results of a marketing campaign which involves randomly splitting a group of people into 4 treatments. One group acts as a control and is shown no advertising, the other 3 groups are ...
1
vote
0answers
110 views

AB Test for Total Revenue: How to take both of conversion rate and average order value into account in my ab-test?

I have some purchase data for two groups of people, for instance: Method Revenue ------ ------- A 12 A 0 A 0 A 4 A 0 B 0 B 8 B ...
1
vote
1answer
66 views

AB testing and hypothesis testing

I have an entire market for which I plan to run two offers. I randomly split the market 50/50. All of Group A receives offer 1 and all of group B receives offer 2. I want to compare proportions of ...
1
vote
1answer
134 views

A/B testing for binary classification tasks

I work on project where each user has an IoT sensor and an ML algorithm tries to identify a given situation. The user carries the sensor 24/7 but I split the time into 'sessions'. For each session, I ...
0
votes
0answers
30 views

Confidence interval in Chi squared testing

How to calculate confidence interval when performing a Chi-Square significance testing? For reference where it's used - http://www.evanmiller.org/ab-testing/chi-squared.html Can we apply the same ...

1 2 3 4 5