Questions tagged [ab-test]

A/B testing, also known as split or bucket testing, is a controlled comparison of the effectiveness of variants of a website, email, or other commercial product.

Filter by
Sorted by
Tagged with
0
votes
0answers
13 views

How plausible are RCTs/AB tests?

Suppose that I wish to quantify the efficacy of a drug compared to a control. I proceed to construct my RCT (using sample sizes which guarantees a particular certainty and significance level). I also ...
2
votes
2answers
30 views

What's the drawbacks of choosing a high statistical power and low significance level?

I want to perform an AB test for two products (a control and a variant). I want to detect a 20% uplift with a very high certainty i.e., statistical power and a very low probability of a type I error (...
1
vote
2answers
69 views
+50

A/B testing segments post-analysis

have the data of an A/B testing experiment. We can assume that the power analysis has been done correctly, so both sets, control and treatment had the necessary samples to find stat sig results. The ...
1
vote
0answers
31 views

How to best evaluate A/B Test result for statistical and practical significance holistically?

I am working on evaluating an experiment or A/B test but I am fairly new to it. In that process, I am trying to interpret which post-test metrics are significant and the methodology to decide that. ...
1
vote
1answer
34 views

Is this a statistical test ? what is the experimental unit? and how to treat dependency between observations?

I'm trying to understand which statistical test to use and what is my experimental unit? The problem: We have a budget algorithm that moves the budget between 10 different ad campaigns and tries to ...
0
votes
1answer
27 views

AB Testing / Experimentation: ratio estimator based on parameter estimates of a regression model

Assume we run an experiment / AB test, hence we have a treatment $D$, exogenous covariates $X$ and outcome variable $Y$. A standard regression estimator then identifies the treatment effect $\tau$ as ...
6
votes
3answers
531 views

Is "Permutation Test" sufficient for a/b testing?

I read the "Practical Statistics for Data Scientists" book which is written by Peter Bruce and Andrew Bruce. In this book, the hypothesis testing is only realized with permutation test. If I ...
0
votes
1answer
27 views

What sample size is good enough for this AB Test experiment?

I am new to AB testing/marketing data science. But would like to understand how to approach this problem i have. I have a model which predicts which customers are likely to be Families. We want to ...
1
vote
0answers
22 views

Stopping an A/B test by monitoring the convergence of the metric of performance

I recently joined a company running a lot of A/B tests with many bad practices. Although I can explain why most of them are flaws, there is one that I can't really wrap my head around. Let's use an ...
3
votes
3answers
429 views

Causal inference where potential outcome is somehow "violated"?

The fundamental problem of causal inference says that only one potential outcome is observed for each unit. What happens if both outcomes from control and treatment can be observed? Can we still make ...
1
vote
1answer
18 views

Conditional Average Treatment Effects

Suppose I am interested in understanding the effectiveness of two type of medicine, A and B. In the usual setting, each patient is considered an experiment unit, and is assigned one of the medicine. ...
0
votes
0answers
15 views

Why does redrawing a sample of a binary variable split into two groups generate different distributions according to prop.test()?

I was working on some power calculations and stumbled upon results I don't understand. Say we have a sample of a binary variable a, and the sample probability of success is p. We randomly split the ...
1
vote
0answers
15 views

How to Build a Model with Correlation / Statistical Dependency for Bayesian A / B Testing

I use the Beta Binomial model for A/B testing. I wonder if there a way to build a model in PyMC which models correlation between the conversion rate of group A with ...
0
votes
0answers
9 views

How to optimally choose winsorization thresholds for different metrics in large scale A/B testing platform

I work on our A/B testing platform where we have implemented one-sided winsorization broadly across all continuous variables (capped at 95th percentile). While that's a common cut-off, some of our ...
0
votes
1answer
47 views

Causal Inference where the treatment assignment is randomized

I have mostly worked with Observational data where the treatment assignment was not randomized. In the past, I have used PSM, IPTW to balance and then calculate ATE. My problem is: Now I am working on ...
1
vote
1answer
25 views

Learn Optimal Parameters of Beta Distribution for Bayesian A/B Test

What are the known / efficient methods to learn the parameters of a Beta Distribution in the context of A/B testing? Is it a reasonable starting point to learn from the history of the base (Let's say ...
0
votes
0answers
15 views

A/B test contingency table splitting in multiple ways....risk of false positives?

I have data for an A/B test conducted for an online store. All the variables are categorical. For every unique visitor the following were recorded: ...
1
vote
1answer
32 views

Purpose of reaching power analysis-specified sample size when effect is already detected in A/B test?

We are performing an A/B test involving web site activity, where samples in both our control and variant groups arrive gradually, day by day. Before we began our experiment, we did a power analysis on ...
0
votes
1answer
48 views

How to solve this AB test question?

Assume you were tasked with improving the listing conversion rate on xx.com. You build an A/B test with 2 variants. Control: Listings page has a button that reads “Reserve a Room” Test: Listings page ...
0
votes
0answers
40 views

How to decide the right sample size in A/B testing

I'm working on a price optimization problem where I want to change the price of a product and see if the revenue becomes higher. So, I have 40 different stores in my experiments and I split them into ...
0
votes
0answers
19 views

Appropriate Way to Measure the Influence

I have two online marketing channels A & B and have their sales data for the past 24 months, as below. Total sales (conversions) are measured as A+B. B is the cheaper channel and hence my ...
0
votes
0answers
92 views

Does it make mathematically sense to aggregate data in order to reduce variance in statistical significance tests?

Let's consider an e-commerce site. We have an AB test for which we want to measure if the average revenue from treatment A is statistically significant different from B. i.e. the main goal is to ...
6
votes
1answer
330 views

How can I learn statistical a/b testing?

I need to learn statistical a/b testing. I have machine learning and probability background, how can I learn about this topic? Which books should I read? Or maybe someone has article suggestions?
0
votes
0answers
28 views

Calculate sample size for a/b testing of continuous value, how to define the expected mean difference?

i want to conduct power analysis for continuous value a/b testing. let say with the treatment, order value will increase. usually i use stats calculator to calculate sample size by only giving total ...
1
vote
0answers
34 views

Does the Bonferroni Correction apply across metrics/KPIs?

Say I am running an A/B test with three total variants and two treatments (Control, T1, and T2). I am also calculating multiple KPIs--say two KPIs (K1 and K2)--for this test. If I apply the ...
1
vote
1answer
118 views

Bayesian A/B testing and decision metrics

Say I need to test two different product features ({existing/control: blue} vs {new/treatment: red} font on webpage, for example), and need to boil my analysis down a to a single go/don't go criteria ...
0
votes
0answers
16 views

A/B testing with seasonality and growth trends?

I'm prepping for an interview with a tech company where the role is largely focused on A/B testing product features. In a vacuum, this seems like simple hypothesis testing. But when I take other ...
2
votes
1answer
106 views

Two Sample Z-test with Non - normal distributions (A/B testing)

I am doing my first steps in statistics and your help will be much appreciated. Sorry for not providing any data, since this is just a made-up exercise that I could not quite find the answer for. Let'...
1
vote
0answers
33 views

Bootstrap Hypothesis Testing Sample Size

I want to use bootstrap to test a hypothesis. I can't figure out how to find out the minimum possible sizes of control and experiment groups. Can you suggest what methods exist for this?
1
vote
1answer
26 views

A/B testing-question on ad placement

I need help on building the thought process to get to the answer to the following question: "You want to test which of the two ad placements on your website is better. How many visitors and/or ...
0
votes
0answers
28 views

Help understanding results of A/B Testing Power Analysis (lift on mean)

Background: I am not a statistician or a mathematician, but I am always willing to learn and do it quite fast. I have taken over (for the moment) the power analysis calculation for our A/B Experiments ...
0
votes
0answers
20 views

When conducting AB testing on binary dependent variable, should a t test being used or logistic regression?

I have simulated ab testing result that is intended to test by changing layout of website,does it increase likelihood to purchase or not(0/1 binary). I tried both logistic regression and t test for it,...
0
votes
1answer
72 views

How to check for novelty effect in AB testing?

I have a simulated ab testing result where the target variables is binary - purchase or not. And I have new and old as customer type. How do I check for the novelty effect? Can I break them into 2 ...
1
vote
1answer
94 views

ABC test/ hypothesis testing with multiple comparisons

I need to do a hypothesis test with three versions (one control version and two test versions). From what I understand I can just run a t-test (or z-test) for each test version vs control, and then ...
0
votes
0answers
43 views

How can we do a test verification for a non-normal distribution?

I know that if we have a normally distributed distribution we can verify whether it's significant with the rates and our normal approximation. For instance with the following segments: ...
0
votes
1answer
37 views

Multivariate Testing

I have an ab test with two factors (color of image and size of image). Is the correct methodology to do these steps (assuming I want to do a full factorial): Calculate the f-stat for the control (...
2
votes
1answer
45 views

Doing a t-test on a normally distributed difference of two non normal distributions

I have two populations of donations (segment 2 people were subject to a different form): ...
2
votes
1answer
65 views

How to calculate z-score of binomial values for an ab test

I ran an ab-test and now I'm trying to calculate the pvalue. The test and control are binomial and these are the values To get the variance, I used the variance formula from here https://stattrek....
0
votes
0answers
88 views

Binomial AB Test - binomial test vs normal approximation

I want to run an ab test on two groups of people If I use the binomial test I get a p-value of 0.007355 ...
1
vote
2answers
108 views

What difference between Mann-whintey U-test and Kolmogorov-Smirnov test on truncated log normal distributions?

I have two populations who have been exposed to two different websites that should bring them to donations: one with a progress bar that pushes them to give (B, segment 2) and the other not (A, ...
3
votes
3answers
98 views

T-test states difference of donation is significant when Z-test claims not, what method to use?

I have two populations who have been exposed to two different websites that should bring them to donations: one with a progress bar that pushes them to give (B, segment 2) and the other not (A, ...
0
votes
1answer
157 views

How to compare conversion rates from AB test when users visit multiple times [duplicate]

I've been asked to compute the minimum sample size for an AB test that my company is conducting for a tweaked recommender system. We will choose two groups of users, one of which sees only the A ...
1
vote
1answer
38 views

AB test with interaction between observations in the two groups

Suppose I am the owner of an e-commerce store. I'd like to design an AB test to measure the average effect of discounts on sales. I am not able split user traffic so that two users see different ...
0
votes
0answers
56 views

Calculating minimum sample size for a bootstrapped a/b test

When leveraging bootstrapping for a/b testing, is it necessary to calculate a sample size required to measure a minimum detectable effect at some specified significance level? If so, how? Context: I'...
2
votes
1answer
61 views

Mann Whitney U versus bootstrapped sampling distribution of difference between means

Context: I want to run an AB test to measure the effect of an algorithmic change and whether the effect is statistically significant. My north star metric, the CPC (cost per click), is not normally ...
8
votes
5answers
2k views

Why do I need statistical power for AB testing if my results are significant?

I have been told that I need both significance and power for my AB results to be valid. I researched a lot for this and the above statement is not making sense. I get that we need high enough power to ...
2
votes
1answer
206 views

A/B test sample size formula confusion

I was trying to understand the math behind some commonly used calculators and formulas for A/B tests and it seems that there might be some variations. Ideally, I would like to understand how each of ...
0
votes
0answers
15 views

Can I subtract two uplifts in A/B Test?

I have a A/B/n test with a control group, T1 with say a change A, T2 with say change B, and T3 with ABC, To understand impact of A, I can calculate incrementality of T1 wrt C. => Lift 1 To ...
0
votes
1answer
39 views

Defining test- and control groups for geo-test with balanced covariates and group size

I'm currently designing an A/B-Test on a geo level, where the test- and controlgroup will be made up of various (US) States. Each state has a few specific characteristics (e.g. mean income, ...
0
votes
0answers
27 views

What is the right approach to bucket users for algorithms with different coverage for A/B testing

I've couple of recommendation algorithms that I want to A/B test. Algorithm A has 90% user coverage and algorithm B has 95% user coverage. That means if the algorithms are asked to provide ...

1
2 3 4 5
7