Linked Questions
17 questions linked to/from Why does collecting data until finding a significant result increase Type I error rate?
2
votes
1
answer
122
views
Changing sample size during experiment? [duplicate]
Will the significance and power levels hold constant if I don't see statistically significant results and decide to wait to allow for more samples to be collected?
6
votes
0
answers
95
views
Is it OK to prolong a non-significant A/B test? [duplicate]
Background: We know from this article that ending an A/B test early due to "significant" results is a mistake.
Question: But what about when a test runs for the desired time period and shows ...
41
votes
2
answers
6k
views
Am I creating bias by using the same random seed over and over?
In almost all of the analysis work that I've ever done I use:
set.seed(42)
It's an homage to Hitchhiker's Guide to the Galaxy. But I'm wondering if I'm ...
17
votes
2
answers
5k
views
Why is it wrong to stop an A/B test before optimal sample size is reached?
I am in charge of presenting the results of A/B tests (run on website variations) at my company. We run the test for a month and then check the p-values at regular intervals until we reach ...
16
votes
3
answers
1k
views
Some questions about statistical randomness
From Wikipedia's statistical randoness:
Global randomness and local randomness are different. Most
philosophical conceptions of randomness are global—because they are
based on the idea that "in ...
19
votes
2
answers
675
views
How to calculate average length of adherence to vegetarianism when we only have survey data about current vegetarians?
A random population sample was surveyed. They were asked if they eat vegetarian diet. If they answered yes, they were also asked to specify how long they’ve been eating vegetarian diet without ...
5
votes
2
answers
122
views
Likelihood of rejecting a fair coin (repeated significance testing)
Suppose I have a fair coin and I flip it numerous times, testing after every time using Pearson's $\chi^2$ test of fit to fairness. What is the likelihood that I will, at some point, reject that the ...
3
votes
1
answer
368
views
data peeking and increasing sample size
Suppose I have designed an experiment to test $D>0$ where $d>0$ is effect size of $D$ with confidence level $0.05$ and power $0.8$. Say I find that I need $2N$ experimental units to get the job ...
2
votes
1
answer
106
views
Does p-hacking affect the Type II (False Negative) error rate?
I know that p-hacking (also know as data dredging) increases the Type I error rate as exemplified by this XKCD example.
My question is whether it influences the Type II error rate. I'm not sure ...
2
votes
0
answers
226
views
Stopping an A/B test by monitoring the convergence of the metric of performance
I recently joined a company running a lot of A/B tests with many bad practices. Although I can explain why most of them are flaws, there is one that I can't really wrap my head around.
Let's use an ...
4
votes
0
answers
137
views
What's the probability the cumulative average of multiple gaussian variables to exceed a certain value?
Consider a number $n$ of independent, normally distributed variables $Y_1$, $Y_2$, ..., $Y_n$.
Consider also the $n$ variables defined by the average of the last $h$ terms, $X_h = \frac{\sum^{n}_{k = ...
3
votes
0
answers
132
views
A/B test results insignificant
If I run an A/B for the recommended duration which was 2 weeks and after that duration, the test results are not statistically significant at 95% significance level. If the PM comes back to me and ...
2
votes
0
answers
52
views
Hypothesis Test for a Linear Combination of Three or More Probabilities
Suppose that you have a parameter w. As you change the value of that parameter, the probability of a given event occurring increases or decreases. I want to use the Method of Finite Differences to ...
0
votes
0
answers
48
views
Why is chi squared for paired coin tosses so sensitive to slight deviations from 1:2:1?
I've been using googlesheets to calculate and understand how chi squares work in paired coin tosses as a model for pairing of alleles in Punnet squares. I used the randbetween(0,1) function to ...
2
votes
1
answer
25
views
Combining results of two studies due to incomplete data
I am doing a study looking at the effects of a certain virus in rats. We are looking at certain concentrations of bio markers in the blood. I came into this project part way through and no power ...