Linked Questions

2
votes
2answers
7k views

interpreting confidence intervals in t.test

I'm trying to run a t-test using t.test() but am confused about the p-value and confidence interval. Looking at the output I see a p-value of ~0.03. That leads me ...
1
vote
2answers
466 views

Why is it necessary to give a-priori a threshhold for p-Values?

Let's say the results for an experiment give a p-Value of 0.0354234. Why is it necessary to fix a threshhold before doing the experiment and than report significance? Why do I not simply report the p-...
0
votes
2answers
129 views

Do probabilities for selected type I error **rates** (alpha's) have confidence intervals?

In a journal article, there can be dozens of statistical tests that are unrelated, and, the more tests there are, the greater the likelihood that the probabilities for one of those tests for a given ...
0
votes
2answers
6k views

Why is smaller the p-value, larger is the significance? [duplicate]

I am trying to understand the concept of p-value. Wikipedia mentions: The p-value is defined as the probability, under the assumption of hypothesis $H$, of obtaining a result equal to or more ...
19
votes
1answer
21k views

Is Benjamini-Hochberg correction more conservative as the number of comparisons increases?

How conservative is Benjamini-Hochberg multiple testing correction relative to the total number of comparisons? For example, if I have a list of 18,000 features for two groups and I perform a Wilcoxon ...
6
votes
1answer
235 views

Is the p-postulate (equal p-values provide equal evidence against the null) true?

The p-postulate is the notion that equal p-values provide equal evidence against the null hypothesis. Wagenmakers et al (2008) write: If p-values truly reflect evidence, a minimum requirement is ...
5
votes
1answer
15k views

Can p-value be greater than 1?

Say I perform Student's t-test: can the p-value be greater than 1? The reason I ask this is that, some say that the p-value is the probability that we will have a Type I error and I would like to ...
5
votes
1answer
2k views

What is an F test?

What is an F test and what does it show? Also what is an alpha, and how do I evaluate the p-value?
4
votes
1answer
79 views

Am I taking crazy pills here or this A/B testing tool's documentation dead wrong?

In the documentation here for a popular A/B testing they say, For example, if you run an A/B testing experiment with a significance level of 95%, this means that if you determine a winner, you ...
3
votes
1answer
195 views

Suppose I have a Cramér-von Mises $p = 0.99$ and Chi-squared $p = 0.88$ of a distribution being Student's-$t$. What can I say?

Suppose I have a Cramér-von Mises $p = 0.99$ and Chi-squared $p = 0.88$ of a distribution being Student's-$t$. What can I say? What can I say about the underlying distribution's being a Student's-$t$...
3
votes
1answer
1k views

Can we calculate the probability that a null hypothesis is true, in general?

I have a very simple question for Hypothesis testing. Is there a way to calculate the probability that the null hypothesis is false? Since in the literature they're always talking about the ...
2
votes
1answer
153 views

Is my interpretation of the p-value and confidence interval correct?

Background: let's say I have a website with a big buy button on it and I preformed an A/B testing with the following result: Version A (control): conversion rate 5% Version B (variation): ...
2
votes
1answer
4k views

FWER, FDR and multiple comparisons for Beginners

Wikipedia: "In statistics, family-wise error rate (FWER) is the probability of making one or more false discoveries, or type I errors, among all the hypotheses when performing multiple hypotheses ...
1
vote
1answer
51 views

Interpretation of the D-statistic from 2-sample KS-tests

I've been using the two sample Kolmogorov-Smirnov test to compare observations from 2018 vs observations from 2019. I heard from my co-worker that you can interpret the D-statistic as the magnitude ...
1
vote
1answer
414 views

Confusion regarding p-values and false discovery rate

I get stuck with understanding of the following two statements (from Wikipedia on p-values): The p-value is the probability of obtaining at least as extreme results given that the null hypothesis ...

15 30 50 per page