Questions tagged [sequential-analysis]
Hypothesis testing without a fixed sample size.
146 questions
20
votes
4
answers
2k
views
How could one develop a stopping rule in a power analysis of two independent proportions?
I am a software developer working on A/B testing systems. I don't have a solid stats background but have been picking up knowledge over the past few months.
A typical test scenario involves comparing ...
- 703
16
votes
0
answers
1k
views
Understanding Sequential Probability Ratio Test (SPRT) Likelihood Ratio
I am a software developer looking to develop an alternative for the simple hypothesis testing scheme described here. In short, the test works as follows:
Two URLs are compared for their ability to ...
- 703
14
votes
5
answers
2k
views
Determining sample size before starting an experiment or run the experiment indefinitely?
I studied statistics years ago and have forgotten it all so these may seem like general conceptual questions than anything specific but here is my issue.
I work for an e-commerce website as a UX ...
- 241
14
votes
3
answers
405
views
Is dynamically increasing the sample size okay, if stated a priori?
I'm about to do a study about the merits of one stimulus compared to another with a within-subject design. I have a permutation scheme that is designed to reduce order effects of some parts of the ...
- 805
13
votes
2
answers
1k
views
Are bayesian methods inherently sequential?
Ie, to do sequential analysis (you don't know ahead of time exactly how much data you will collect) with frequentist methods requires special care; you can't just collect data until the p-value gets ...
- 545
13
votes
2
answers
1k
views
Adjusting the p-value for adaptive sequential analysis (for chi square test)?
I wish to know what statistical literature is relevant for the following problem, and maybe even an idea on how to solve it.
Imagine the following problem:
We have 4 possible treatments for some ...
- 21.9k
12
votes
6
answers
4k
views
Why isn't a confidence level of anything >50% "good enough"?
I took a graduate-level stat course so I feel really dumb asking this, but I'm having trouble with the meaning behind 95% confidence. (Please be nice to me.)
I currently running an A/B test on two ...
- 319
11
votes
2
answers
2k
views
High autocorrelation when taking the L-th order of difference of a sequence of independent random numbers
To explain this question in more detail, I'll first elaborate my approach:
I simulated a sequence of independent random numbers $X = \{x_1,...,x_N\}$.
I then take $L$ times the difference; i.e. I ...
- 547
10
votes
2
answers
385
views
Updating a Bayes factor
A Bayes factor is defined in Bayesian testing of hypothesis and Bayesian model selection by the ratio of two marginal likelihoods: given an iid sample $(x_1,\ldots,x_n)$ and respective sampling ...
- 108k
8
votes
1
answer
599
views
How to distinguish between two biased coins
I have two very biased coins:
The first coin (C1) lands on heads 95% of the time.
The second coin (C2) lands on heads only 1% of the time.
One of the coins has been selected and I want to determine ...
- 105
7
votes
1
answer
376
views
Stopping rules in group sequential designs
I have read that in group sequential designs, having a maximum of $k$ stops, for the $j$-th stop $j < k$ there are 4 bounds chosen $a_j,b_j, c_j, d_j$ such that the three decisions are considered. ...
- 64.8k
7
votes
1
answer
916
views
Explanation for the thresholds in the sequential probability ratio test
Recently I have learned about sequential analysis especially sequential probability ratio tests (after I have struggled a lot with cumulation of alpha - errrors). See also this question Sequential ...
- 10.5k
7
votes
1
answer
366
views
Testing for Benford Law in real time
Suppose I have data of a certain quantity $X$, given by $x_1,...,x_n$. Now, I take the first digit $d_i$ of each quantity $x_i$, and I want to study the relationship between the empirical distribution ...
- 171
7
votes
0
answers
221
views
bias of an estimator when using stopping rules
Consider the setting where $X_1,X_2,...$ are i.i.d. real-valued random variables with $\mathbb{E}[X_i] = \theta$ and let the random variable $\tau$ be an associated stopping time. I'm wondering what ...
- 1,268
6
votes
1
answer
1k
views
What are disadvantages of "Sequential analysis"
Sequential analysis involves performing sequential interim analysis till results are significant or till a maximum number of interim analyses is reached.
Sequential analysis sounds appealing ...
- 10.2k
6
votes
1
answer
276
views
Equivalence of Contextual Bandit formulations
I find two different types of formulations for Contextual Bandits:
Definition 1: In a contextual bandits problem, there is a distribution $P$ over $(x,r_1,...,r_k)$, where $x$ is the context, $a \in [...
- 896
6
votes
1
answer
797
views
Speed up web a/b tests with sample size checkpoints
Before starting an a/b test with a control and one experimental route, I can calculate a required sample size based on conversion rate estimates for both routes. I can get a good estimate of the ...
- 161
5
votes
1
answer
4k
views
How many samples do I need to estimate a gaussian distribution?
Background: I'm comparing the performance of two algorithms. The data is noisy so the results tend to form a nice normal distribution.
I'd like to estimate the mean with a high confidence: up to 1% ...
- 51
5
votes
1
answer
1k
views
Sequential testing for a Bernoulli variable
Suppose $T$ is a Bernoulli random variable, such that $T = 1$ with probability $s$, $s$ is not known.
I am interested in checking hypothesis $H_1$ that $s \neq 1/2$ ($H_0$ is that $s = 1/2$). One way ...
- 415
5
votes
1
answer
79
views
How to pick the winner in the "Play the Winner" treatment assignment scheme (Urn model)
"Play the winner" is an intutitive procedure whereby successful treatments in earlier trials have a higher probability of being assigned in later trials. Quoting https://online.stat.psu.edu/...
- 5,447
5
votes
1
answer
165
views
Earliest significance in A/B test
I'm working on a problem/assignment where I'm given some activity logs from a website where an experiment was conducted. I'm asked to determine the earliest time when a statistically significant ...
- 435
5
votes
1
answer
594
views
Sequential testing of multiple probabilities (binomial variables)
Task:
I have e.g. 1 control group and 7 experimental groups which are treated differently. All groups have a dependent binomial variable: 1 = success, 0 = no success.
I want to conduct sequential ...
- 51
5
votes
1
answer
535
views
How to efficiently expand a simple random sample after population growth?
I have a simple random sample (SRS) $S$ of size $n$ drawn from a population $D_1$ containing $N_1$ members. A new set $D_2$ with $N_2$ members is added to the population. I would like to create a ...
- 721
5
votes
2
answers
777
views
Sequential updating of a Bayesian regression model
I'd like to perform Bayesian regression of $Y$ on $X$. i.e. estimate a posterior $p\left(\theta\mid\mathcal{D}\right)$ of some regression parameter $\theta$ given a dataset $\mathcal{D}$ of pairs of ...
- 451
4
votes
1
answer
99
views
Is the given procedure SPRT?
In this question, I tried using the fundamental property of the Wald Sequential Probability Ratio Test (SPRT): $P( N = \infty )=0.$
For this I calculated $p_1 = 1- P(-1.5 < S_1 < 2.5 )$, that ...
- 387
4
votes
1
answer
139
views
Can the Bayesian but not the frequentist "just add more observations"?
Since the frequentist's p-values are uniformly distributed under the null hypothesis, it is a highly problematic practice to add more and more data to your sample until you find a significant result. ...
- 79
4
votes
2
answers
277
views
Frequentist performance of sequential testing based on predictive power
Suppose we perform a hypothesis test from a random sample $(x_i)_{i=1}^n$, assuming for instance $x_i \sim_{\text{iid}} {\cal N}(\theta, 1)$ and $H_0=\{\theta=0\}$ for simplicity. If the test fails to ...
- 19.7k
4
votes
1
answer
479
views
AB Multi-variate testing - when can I eliminate worst performing variants?
I'm running AB tests on my website homepage, with 8 different variations. You can read about the purpose of the test here if you need to (not essential) - http://westiseast.co.uk/blog/ab-split-testing-...
Chris
4
votes
1
answer
505
views
Performing a Wald's group sequential test in R (for complex hypothesis)
The settings: we have a sequence of numbers generated from a normal distribution with $\mu=1;\sigma^2 = 1 $. And with every new observation collected, we'd like to test for: $H_0:\mu_0=0$ (two-sided)....
- 21.9k
4
votes
2
answers
148
views
How are group sequential analysis, random walks, and Brownian motion related?
Assume that I am planning a clinical trial comparing two groups using a binary outcome. I will do the $\chi^2$ test after 3 equal enrollment intervals: interim test #1 after $m_1$ enrollments in ...
- 426
4
votes
1
answer
318
views
sequential estimators for a proportion
Does anyone have any reference for a sequential estimator for proportions?
I am planning an experiment to measure the probability of a certain event happening and would like to know how many ...
- 363
4
votes
1
answer
189
views
When to stop an ad campaign?
I have a campaign and I am trying to decide whether to terminate or not, with profit maximization in mind. There is cost associated with every time the ad is shown, and a reward once it converts (...
4
votes
1
answer
315
views
Sequential Inference And Evidence (Jaynes 2003): Is it valid? Is it used?
Exploring the work of ET Jaynes, Probability Theory (11th Printing 2013) has led to consideration of the technique he identifies as Sequential Inference (p. 96); where the evidence, in decibels, ...
- 261
4
votes
0
answers
85
views
Sequential hypothesis test for uniform distribution
Suppose I have a sequence of data points $x_1,x_2,\dots$. I want to do a sequential hypothesis test, were the null hypothesis is that each $x_i$ is iid uniform on $[0,1]$, and the alternative ...
- 6,738
4
votes
0
answers
150
views
textbooks/literature/resources for online learning/ time series/ sequential analysis? [closed]
I have read (the contents page of) machine learning introductory textbooks such as Pattern Recognition and Machine Learning, Machine Learning A Probabilistic Perspective, and The Elements of ...
- 16.8k
3
votes
3
answers
195
views
Group sequential Design
Can anyone recommend a paper which provides a nice introduction or summary of group sequential design.
- 1,183
3
votes
1
answer
704
views
Term for when hypothesis testing is conducted repeatedly until an interesting result is found?
What is the term for when, through accident or intent, many hypothesis tests are conducted until an 'interesting' result is found? That is, the experiment (or variations on it) are repeated until a ...
- 303
3
votes
1
answer
93
views
How to parametrize a beta prior for a sequential Bayesian analysis of a proportion?
I'm developing a decision rule for a study using an interim analysis with a Bayesian characterization of the event probability. At a certain point, we will perform a planned stop, and only proceed ...
- 64.8k
3
votes
1
answer
362
views
Does an interim sample size re-estimation increase type 1 error if based on the overall event rate?
Suppose that a sample size is to be calculated for a trial: a presumed effect size summarizes the effect of intervention (relative risk RR $\exp(\theta_1)$), and background gives the rate of events in ...
- 64.8k
3
votes
1
answer
2k
views
Should one adjust analyses if interim analysis is planned for an outcome other than the primary outcome?
It is widely accepted that analyses of clinical trials have to be adjusted if interim analyses are planned in order to decide if early termination is necessary due to evidence for superiority of one ...
- 33
3
votes
1
answer
60
views
Can't reproduce closed form expression from sequential testing
I've been trying to implement and extend some results from the papers "Always Valid Inference" and "Peeking at A/B Tests". The authors provide a closed form expression of the "...
- 41
3
votes
1
answer
193
views
How to test for deviation from a uniform distribution?
I have a random number generator as part of a system that - putatively - randomly selects one of 10,000 drive locations to store an incoming file fragment. Multiple file fragments can fill a drive ...
- 177
3
votes
2
answers
448
views
Sequential experiment: controlled design and metrics
This is a generic question. Let me put forth an example scenario. Say, I had 2 techniques for allocating my daily budget within 4 stocks. Upon allocation, I get the data about the stock's performance ...
- 2,545
3
votes
2
answers
2k
views
Why is this regret a good choice for a multi-armed bandit?
The regret in a multi-arm bandit model is given by
$$\underset{j}{\max}\sum_{t=1}^{T}x_j(t) -G_{A}$$
where $$G_A=\sum_{t=1}^{T}x_{it}(t)$$ is the total reward achieved by the learner, based on an ...
- 2,545
3
votes
1
answer
86
views
Determining Optimal Step Size for Sequential Analysis
Consider the following problem:
Suppose someone wanted to determine the ideal dosage of an epinephrine shot in order for the shot to be successful 90% of the time. The person is also not willing to ...
- 76
3
votes
0
answers
38
views
Techniques for Stopping a Diagnostic Study for Safety
I'm working with a group of physicians who are interested in evaluating a new diagnostic tool for detecting colon cancer in older patients. The gold standard to detect colon cancer is with a ...
- 39.6k
3
votes
0
answers
53
views
prior distribution for iid gaussian, with a known variance
I have been reading Pattern Recognition and Machine Learning by Bishop, and I have a question regarding the prior distribution of an iid Gaussian with known variance.
The relationship $\dfrac{n}{\...
- 9,317
3
votes
0
answers
117
views
Mixed-effects modelling in a sequential analysis design
I'm planning a study and am considering going with a sequential analysis design (as described here: https://psyarxiv.com/9yegd/download?format=pdf). This design allows one to take multiple looks at ...
- 73
3
votes
0
answers
28
views
Advice for conducting the same experiment several times?
I'm very new to Bayesian statistics, but I've found myself in a situation where they might be able to help. I'll be running the same experiment repeatedly and analyzing the data every few months. The ...
- 283
3
votes
1
answer
119
views
How to reconcile results from many incremental hypothesis tests?
I'm considering a series of controlled experiments (e.g. A/B tests) measuring the performance of a system. Each time a change to the system is found significant (via statistical inference) the test ...
- 1,292