Questions tagged [ab-test]

A/B testing, also known as split or bucket testing, is a controlled comparison of the effectiveness of variants of a website, email, or other commercial product.

Filter by
Sorted by
Tagged with
0 votes
0 answers
13 views

Stat sig results on underpowered experiment

I ran an experiment that was powered enough to detect lifts of 1% on my 2-sample test. Make it really simple: I have two normal distributions and want to compare the means, so I ran a 2-sample t-test ...
user avatar
  • 87
0 votes
1 answer
28 views

Assessing the accuracy of a particular A/B testing design methodology

I commented to a colleague that we could improve our measurement of treatment effects by ensuring proper randomisation into treatment and control groups by balancing the treatment and control groups ...
user avatar
0 votes
0 answers
16 views

A/B testing when there is a feedback loop

I am experimenting with changing label value (target) for a model that we have in production. We used to cap the target variable, and my new model will release the cap. The main point about our ...
user avatar
  • 279
0 votes
0 answers
6 views

Understanding the switchback experiment process end to end(from setup to analysing the results) [closed]

Can anyone help me on understanding the switchback experiment process end to end(from setup to analysing the results). I read few blogs on it but i am not able to understand it properly. Also please ...
user avatar
0 votes
0 answers
15 views

Are variances in A/B test the same? [duplicate]

When running an A/B test in python with scipy stats, we haven argument call: equal_var. More info here: https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ttest_ind.html But basically it ...
user avatar
  • 87
3 votes
2 answers
48 views

What to do if you Double count in experimental design?

I have to run A/B/n tests for a subscription service. Generally computing metrics for this situtation is ok: For example, coversion rate experiments. We have 1000 prospects in a group, and (say) 89 ...
user avatar
  • 121
3 votes
1 answer
80 views

is ab test valid for concluding causal relation, when analysis unit differs from randomization unit?

In a typical A/B test, the randomization unit is user level, sometimes the analysis unit may be page/visit level, like a cluster randomization experiment. In this situation, the iid assumption doesn't ...
user avatar
  • 96
0 votes
0 answers
50 views

Ratio Confidence Interval for Proportions

Wondering if Fieller's theorem is used for the 'ratio' value in the compare argument in ...
user avatar
0 votes
1 answer
32 views

Two Sample Independent Test for Proportions (z test or t test)

Context: I have two independent samples that I want to compare for equality through an AB test. The metric being evaluated is binary: user clicks or not (proportions). Question: I keep reading that z ...
user avatar
0 votes
0 answers
15 views

Which Hypothesis Test for AB test with 4 variants

Disclosure: Stat noob trying to understand the basics of ab testing Context: I'm running an AB test with 3 treatment test to see which one yields the highest click thru rate. Q: What is the recommend ...
user avatar
3 votes
3 answers
83 views

P-value under 5% but power under 80%

Im trying to understand how to accurately report results on AB tests If my type 1 error is 5% and type 2 is 20% , can we have instances where the p-value is < 0.05 and power is < 0.8? If so, how ...
user avatar
0 votes
1 answer
51 views

confidence interval: A/B test - unable to decide what to use

I am new to A/B test; I am interested to generate confidence interval in python language. At the moment I have ratio (proportion), for instance, ratio of add to cart to the total sessions in control ...
user avatar
0 votes
0 answers
13 views

AB Testing with Time Locked Metric of Interest

had a general AB test question surrounding how to best measure time locked performance of a metric of interest. The experiment basis is pretty simple: on a hosting website, users click on a button to ...
user avatar
  • 311
0 votes
1 answer
40 views

Sample size study not reflecting findings in R simulation

I have a binomial response variable (a sort of conversion rate for a search mechanism) and just one independent variable, which is the "type" of the search bar template. To illustrate, my ...
user avatar
0 votes
0 answers
32 views

Which hypothesis testing should I use for pageview/user during ab test?

I am currently running an experiment on a website, there are control and 1 variant. The variant changes one of the features of the website. Besides calculating the click-through-rate(CTR) of that ...
user avatar
1 vote
1 answer
72 views

Difference in means vs OLS regression coefficients

Suppose I have a data set where each row represents a test subject. There's a dependent variable (y) and two binary columns (x1, ...
user avatar
0 votes
0 answers
15 views

Hypothesis test for classification model

I have a model that outputs 0 or 1 for interest/not-interest in a job. I'm doing an A/B/C test comparing two models (treatment groups) and none (control group). ANOVA for hypothesis testing and t-test ...
user avatar
1 vote
1 answer
29 views

What is the general right call to make from a marginal difference in A/B test results in recommender system?

This was one of the business-related questions from my technical interview last week for a data science position in a recommender system team at a search engine company focusing on advertisement ...
user avatar
  • 221
1 vote
1 answer
66 views

Bootstrap for two-proportion test with grouped data

I have a dataset from an AB test for clickthroughs on a website. We randomly divided users into A and B groups and counted an observation each time a user viewed the webpage. Each observation is one ...
user avatar
  • 137
0 votes
0 answers
29 views

Bayesian A/B testing with normal conjugate model for huge (non-normal) sample sizes

I'm running an A/B test with 100k+ users per group. It consists of a lot of different metrics, some continuous, some counts... they are highly skewed, with a long tail. I'm mostly interested in the ...
user avatar
2 votes
1 answer
34 views

Sample size calculation for precision at $k$: a/b-testing a recommender system

I would like to conduct an online experiment to compare two different versions of a recommender system. The system returns a list of $r$ ranked recommendations. I would like to evaluate the ...
user avatar
1 vote
0 answers
39 views

AB Testing: Possible to use bootstrapping to obtain the probability that each variant is optimal?

I've recently thought of an extremely simple way to interpret an AB test. The only issue is, I'm not sure it's actually a valid approach — I do not have formal training in statistics, so this might be ...
user avatar
1 vote
1 answer
32 views

AB Test, Experimental Design based on Machine Learning Output

Looking for the best way to set up an AB test. Scenario is that the online product I am working with has logged in users that pay a monthly fee, and they can upgrade to get access to more features. ...
user avatar
2 votes
2 answers
68 views

A/B test with a result of another deep learning analysis with 90% accuracy

I am planning to conduct a A/B test with data obtained through a deep learning algorithm. Say, I got a binary classification dataset through machine learning with about 100k rows classified into yes, ...
user avatar
  • 53
1 vote
1 answer
126 views

a/b testing with two different treatment groups

I want to run an A/B test with two machine learning models M1 and M2 (called treatment groups ...
user avatar
0 votes
1 answer
48 views

How to do conclude a treatment works when sample size is small with large STD and small effective value

I have a data which population is 3000; mean is 16; stdev is 44; I want to have a test case to check if the mean can be improved to 16.5 or not. After the calculation, I was told the required sample ...
user avatar
  • 101
0 votes
1 answer
68 views

How to power a strong superiority hypothesis in an A/B test

Suppose I would like to perform an A/B Test, where it only makes sense to implement the "treatment" over the "control" if the "treatment" is atleast 2% better than the &...
user avatar
  • 341
3 votes
1 answer
89 views

Validation of Bayesian Hypothesis for AB test

I have been following this methodology to implement a Bayesian A/B testing with Python on a new search engine feature that helps users to find products more accurately. To be more accurate, I split my ...
user avatar
  • 31
1 vote
0 answers
23 views

A Proper Conjugate Model for A/B Test for Revenue per Click (RPC)

What would be a proper Conjugate Posterior model for Earning / Revenue per Click in A/B test? The data is the total number of visitors and the total revenue per day per variant (A and B). What are the ...
user avatar
0 votes
1 answer
40 views

How to determine power using standard error and effect size?

I was reading this article published from airbnb around gaurdrail metrics. In it, they discuss this power guardrail they derived but it's very unclear how they went about it. How are they determining ...
user avatar
  • 1
0 votes
0 answers
45 views

Controlling daily variation in A/B test

I want to run an A/B test over, hopefully, as shortest time as possible! I have severe variations in the metric over the week which inflates the variance of metrics and hurts the power a lot! I ...
user avatar
  • 325
0 votes
0 answers
22 views

What is the optimal AB testing strategy for multiple variants?

I've come up against an interesting problem. Let's say I work for a subscription-based business that uses a set of in-app prompts to try to get users to purchase a paid subscription. There are three ...
user avatar
1 vote
0 answers
42 views

Interpretation of A/B Test with 3 Variants

We are currently having an interresting discusison about a recent A/B Test we ran and i'd very much appreciate your thoughts on it: We recently ran a A/B/C Test regarding the pricing of same day ...
user avatar
  • 11
0 votes
1 answer
72 views

How to evaluate AB test for determining price elasticity?

Given a standard AB test setup on an e-commerce platform with users randomly seeing same product priced either with control price or a variant, how does one correctly assess the results? Say, after a ...
user avatar
  • 1
1 vote
0 answers
25 views

skewed results due to omitting intersecting populations

A randomizer assigns visitors to my website to one of two groups, for an A/B test, e.g. at 25% and 75% probabilities for A and B respectively. It's supposed to assign them consistently, so that if ...
user avatar
  • 11
2 votes
1 answer
257 views

Inconsistent result in proportion test in R

I have some problems understanding the results of this online experiment. I sent my website users 2 types of messages that can result in a subscription or not. The users can belong to two different ...
user avatar
0 votes
0 answers
24 views

How plausible are RCTs/AB tests?

Suppose that I wish to quantify the efficacy of a drug compared to a control. I proceed to construct my RCT (using sample sizes which guarantees a particular certainty and significance level). I also ...
user avatar
  • 1
2 votes
2 answers
49 views

What's the drawbacks of choosing a high statistical power and low significance level?

I want to perform an AB test for two products (a control and a variant). I want to detect a 20% uplift with a very high certainty i.e., statistical power and a very low probability of a type I error (...
user avatar
  • 21
1 vote
2 answers
180 views

A/B testing segments post-analysis

have the data of an A/B testing experiment. We can assume that the power analysis has been done correctly, so both sets, control and treatment had the necessary samples to find stat sig results. The ...
user avatar
  • 87
2 votes
1 answer
55 views

How to best evaluate A/B Test result for statistical and practical significance holistically?

I am working on evaluating an experiment or A/B test but I am fairly new to it. In that process, I am trying to interpret which post-test metrics are significant and the methodology to decide that. ...
user avatar
1 vote
1 answer
40 views

Is this a statistical test ? what is the experimental unit? and how to treat dependency between observations?

I'm trying to understand which statistical test to use and what is my experimental unit? The problem: We have a budget algorithm that moves the budget between 10 different ad campaigns and tries to ...
user avatar
0 votes
1 answer
46 views

AB Testing / Experimentation: ratio estimator based on parameter estimates of a regression model

Assume we run an experiment / AB test, hence we have a treatment $D$, exogenous covariates $X$ and outcome variable $Y$. A standard regression estimator then identifies the treatment effect $\tau$ as ...
user avatar
  • 263
6 votes
3 answers
715 views

Is "Permutation Test" sufficient for a/b testing?

I read the "Practical Statistics for Data Scientists" book which is written by Peter Bruce and Andrew Bruce. In this book, the hypothesis testing is only realized with permutation test. If I ...
user avatar
0 votes
1 answer
39 views

What sample size is good enough for this AB Test experiment?

I am new to AB testing/marketing data science. But would like to understand how to approach this problem i have. I have a model which predicts which customers are likely to be Families. We want to ...
user avatar
  • 299
1 vote
0 answers
39 views

Stopping an A/B test by monitoring the convergence of the metric of performance

I recently joined a company running a lot of A/B tests with many bad practices. Although I can explain why most of them are flaws, there is one that I can't really wrap my head around. Let's use an ...
user avatar
3 votes
3 answers
453 views

Causal inference where potential outcome is somehow "violated"?

The fundamental problem of causal inference says that only one potential outcome is observed for each unit. What happens if both outcomes from control and treatment can be observed? Can we still make ...
user avatar
  • 301
1 vote
1 answer
54 views

Conditional Average Treatment Effects

Suppose I am interested in understanding the effectiveness of two type of medicine, A and B. In the usual setting, each patient is considered an experiment unit, and is assigned one of the medicine. ...
user avatar
  • 301
0 votes
0 answers
25 views

Why does redrawing a sample of a binary variable split into two groups generate different distributions according to prop.test()?

I was working on some power calculations and stumbled upon results I don't understand. Say we have a sample of a binary variable a, and the sample probability of success is p. We randomly split the ...
user avatar
2 votes
0 answers
37 views

How to Build a Model with Correlation / Statistical Dependency for Bayesian A / B Testing

I use the Beta Binomial model for A/B testing. I wonder if there a way to build a model in PyMC which models correlation between the conversion rate of group A with ...
user avatar
  • 143
0 votes
0 answers
67 views

How to optimally choose winsorization thresholds for different metrics in large scale A/B testing platform

I work on our A/B testing platform where we have implemented one-sided winsorization broadly across all continuous variables (capped at 95th percentile). While that's a common cut-off, some of our ...
user avatar
  • 1

1
2 3 4 5
7