# Tag Info

12

Theory If the autocorrelation is going to have any meaning, we must suppose the original random variables $X_0, X_1, \ldots, X_N$ have the same variance, which--by a suitable choice of units of measure--we may set to unity. From the formula for the $L^\text{th}$ finite difference $$X^{(L)}_i=(\Delta^L(X))_i = \sum_{k=0}^L (-1)^{L-k}\binom{L}{k} X_{i+k}$$ ...

11

I think the concept you are searching for is sequential analysis. There are a number of questions on this site tagged with the term which you might find useful, perhaps Adjusting the p-value for adaptive sequential analysis (for chi square test)? would be a place to start. You could also consult the Wikipedia article here. Another useful search term is alpha ...

7

This is an interesting problem and the associated techniques are have lots of applications. They are often called "interim monitoring" strategies or "sequential experimental design" (the wikipedia article, which you linked to, is unfortunately a little sparse), but there are several ways to go about this. I think @user27564 is mistaken in saying that these ...

6

Say the odds of getting $C_1$ over $C_2$ are, in principle, $1:1$. Then, you flip the coin $n$ times and get $x$ heads. If we call the probabilities of heads $p_1=0.95$ and $p_2=0.01$, then the probability that each coin gives $x$ heads is: $$P(x|C_1)=p_1^x(1-p_1)^{n-x}$$ $$P(x|C_2)=p_2^x(1-p_2)^{n-x}$$ If we use Bayes theorem, we get the odds that the coin ...

5

Rouder (2014) has a nice paper on this (written for psychologists), explaining why sequential testing (so-called data peeking) is fine from a Bayesian perspective. (Paper is freely available online if you do a search for it.) Schoenbrodt et al. (in press) present nice analyses showing how to use sequential analysis with Bayes factors to determine when to ...

5

At what point is the experiment conclusive then? I think this is where the error in thinking is. There is no point at which the experiment can be "conclusive" if you take that to mean "deductively prove causation". When you're doing an experiment that involves a statistical test, you need to make a commitment regarding what evidence you consider to be good ...

4

Presumably the purpose of a recursive equation for Bayes factor would be when you have already calculated the Bayes factor for $n$ data points, and you want to be able to update this with one additional data point. It does seem that it is possible to do this without recomputing the marginals of the previous data vector, so long as the form of the posterior ...

4

As mentioned in the comments, the terminology p-hacking or data dredging are used for such actions of repeating hypothesis tests until through random occurrence a favored outcome or p-value is produced. Cherrypicking the data is also a term for when you pick or disregard data in a way that favors your hypothesis, often deliberately. HARKing is when you form ...

3

This is expected because the differences are not independent of each other. For example, $dX_1(1) \equiv X(2) - X(1)$ is directly proportional to $X(2)$ while $dX_1(2) \equiv X(3) - X(2)$ is inversely proportional to $X(2).$ Because the definitions of consecutive elements of $dX_1$ share elements of $X$ in this inverse way, we expect them to be inversely ...

3

This is more a comment or, at best, maybe a further clue to solve your question, but my reputation doesn't allow me to post comments. I replicated your experiment in Stata using draws from a standard Normal with the following code: clear all set obs 100000 gen t = _n tsset t drawnorm x, n(100000) forvalues i = 1(1)100 { generate Di' = Di'.x } Looking ...

3

If you want the width of your confidence interval to have length no larger than 1 (0.5 on each side) then let's do some simple math. The radius of a confidence interval, as a function of the sample size, is $$\dfrac{1.96 \cdot \sigma}{\sqrt{n}}$$ I'm using the z-statistic here instead of a t-statistic because in the limit, the two are equivalent (I'm ...

3

This is exactly why a clear criterion needs to be defined ahead of trials. As @mdewey indicates there are established methods for periodically evaluating a trial but these all require a clear stopping criteron to prevent any fudging over the decision. Two critical issues are that you need to correct for multiple comparisons and that each analysis is not ...

3

I think you are asking the wrong question here. The question you are asking is about statistical tests; I think the right question is "why is the effect changing over time?" If you are measuring a 0/1 variable for conversion (did they buy at all?) then people who did not buy in an initial session may come back and buy later. This means that the conversion ...

3

It's not that the procedure you describe (keep collecting data until you like the results) does not inflate the type 1 error, if you naively conduct repeated Bayesian analyses, it's that the brand of Bayesian that considers that there is no issue in repeatedly looking at data simply considers type 1 errors an irrelevant concept - and would likely also not ...

3

There's an extensive literature on this you can find easily e.g. via Google scholar. It seems that simulations indicate that sample size reestimation on blinded event rates is fine (see this paper or this one or this one).

3

From a frequentist perspective, there are some clear disadvantages of a sequential analyses. That is, if we are concerned with preserving type I errors, we need to recognize that we are doing multiple comparisons: if I do 3 analyses of the data, then I have three non-independent chances to make a type I error and need to adjust my inference as such. There's ...

2

This approach doesn't have the properties you would have if you fixed the sample size ahead of time. The situation where you look for a particular result while your experiment continues and have some 'stopping rule' (halt your experiment early if a particular situation is achieved) is a version of sequential analysis; see also SPRT. You have to take care ...

2

A bit late, but this might be helpful if somebody stumbles upon this question in the future: Firstly, it is important to distinguish 'group sequential testing' and 'testing after every new observation', which is sometimes referred to as fully sequential testing. In group sequential testing, the null hypothesis is tested after group of observations are ...

2

The reason you don't see notions like a "predefinable alpha error rate" with Bayesian updating is that the update of the posterior often accounts for these effects. If the sole decision that's needed is when to stop, then that can be posed as a Bayesian hierarchical model. If tests need to be done along the way, then, yes, life is more complicated and you'...

2

It might be that the group sequential testing approach is overkill for your application. That is most useful when it is costly to perform interim analyses. If you have very little cost to analysis, then you can just do sequential testing. There are many references to cover the theory of sequential testing. Your case is usually covered as an example ...

2

As the blog you link to mentions, repeated testing does impact significance. Because significance is based on the assumed sampling distribution under the null hypothesis, the sample space changes every time you "peek" at the data. That blog hints your options: Until sequential or Bayesian experiment designs are implemented in software, anyone running web ...

2

This is a sampling size determination problem, check wikipedia on that. In your case, you are estimating a multinomial distribution, for which you might want to check out this old article from JSTOR, or maybe this more recent freely available one.

2

It depends on the purpose of the interim analysis and type of trial. In general, adjustments for interim analyses are potentially needed, if it is desired to control the type I error rate. If the interim analysis is for futility or safety stopping only (i.e. the trial may be abandoned without rejecting any null hypotheses, either because it seems unlikely ...

2

I assume that this is some form of randomized controlled trial and that there is no chance that the unblinded group/data monitoring committee would decide to reject $H_0$ after seeing the interim results (some authors argue to always add an adjustment to give them a criterion and to always pay some penalty for all interim analyses). The main question would ...

2

This area of sequential clinical trials has been explored substantially in the literature. Some of the notable researchers are Scott Emerson, Tom Flemming, David DeMets, Stephen Senn, and Stuart Pocock among others. It's possible to specify an "alpha-spending-rule". The term has its origins in the nature of frequentist (non-Fisherian) testing where, each ...

2

My guess is that this hasn't been really worked out. Most researchers size the study to achieve 0.9 power at the final analysis, relegating interim analyses to just 'hope for the best' in detecting an early, large difference. The early look will have low power because of fewer events and because of the conservative group-sequential cutoff. A full Bayesian ...

2

What's the point of having a futility hypothesis? Government regulations and medical ethics affect the design of medical studies and govern the conduct of clinical trials by describing good clinical practices (GCPs) for studies with both human and non-human animal subjects. Examples and supporting quotes from the articles: "When inferiority meets non-...

2

So it might be helpful to explain in words what is happening in equations $(1)$ and $(2)$ above. These two equations are essentially applying the Law of Total Probability to split up your probabilities into the two terms in each sum, and then applying the (probability) Chain Rule to expand each term into a product of conditional probabilities - essentially a ...

2

Both formulations are equivalent and compatible with standard models of contextual multi-armed bandits, where you assume that you have information available about the environment that does not depend directly of your arm choices. This context, however, can be informative and thus predictive of what rewards / regret you may face based on what arm you pull. ...

1

The integral you want to solve is, $$E(z \mid \mu) = \sqrt{\frac{\lambda}{\pi}} \left[ (\mu_1 - \mu_0) S - \frac{1}{2} (\mu_1^2 - \mu_0^2) T \right],$$ where $$S = \int_{-\infty}^\infty x \exp\left(- \lambda(x-\mu)^2\right) dx,$$ and $$T = \int_{-\infty}^\infty \exp\left(- \lambda(x-\mu)^2\right) dx.$$ I'm using $\lambda = \frac{1}{2 \sigma^2},$ but ...

Only top voted, non community-wiki answers of a minimum length are eligible