30 questions linked to/from Misunderstanding a P-value?
7k views

### interpreting confidence intervals in t.test

I'm trying to run a t-test using t.test() but am confused about the p-value and confidence interval. Looking at the output I see a p-value of ~0.03. That leads me ...
466 views

### Why is it necessary to give a-priori a threshhold for p-Values?

Let's say the results for an experiment give a p-Value of 0.0354234. Why is it necessary to fix a threshhold before doing the experiment and than report significance? Why do I not simply report the p-...
129 views

### Do probabilities for selected type I error **rates** (alpha's) have confidence intervals?

In a journal article, there can be dozens of statistical tests that are unrelated, and, the more tests there are, the greater the likelihood that the probabilities for one of those tests for a given ...
6k views

### Why is smaller the p-value, larger is the significance? [duplicate]

I am trying to understand the concept of p-value. Wikipedia mentions: The p-value is defined as the probability, under the assumption of hypothesis $H$, of obtaining a result equal to or more ...
21k views

### Is Benjamini-Hochberg correction more conservative as the number of comparisons increases?

How conservative is Benjamini-Hochberg multiple testing correction relative to the total number of comparisons? For example, if I have a list of 18,000 features for two groups and I perform a Wilcoxon ...
235 views

### Is the p-postulate (equal p-values provide equal evidence against the null) true?

The p-postulate is the notion that equal p-values provide equal evidence against the null hypothesis. Wagenmakers et al (2008) write: If p-values truly reflect evidence, a minimum requirement is ...
15k views

### Can p-value be greater than 1?

Say I perform Student's t-test: can the p-value be greater than 1? The reason I ask this is that, some say that the p-value is the probability that we will have a Type I error and I would like to ...
2k views

### What is an F test?

What is an F test and what does it show? Also what is an alpha, and how do I evaluate the p-value?
79 views

### Am I taking crazy pills here or this A/B testing tool's documentation dead wrong?

In the documentation here for a popular A/B testing they say, For example, if you run an A/B testing experiment with a significance level of 95%, this means that if you determine a winner, you ...
195 views

### Suppose I have a Cramér-von Mises $p = 0.99$ and Chi-squared $p = 0.88$ of a distribution being Student's-$t$. What can I say?

Suppose I have a Cramér-von Mises $p = 0.99$ and Chi-squared $p = 0.88$ of a distribution being Student's-$t$. What can I say? What can I say about the underlying distribution's being a Student's-$t$...
1k views

### Can we calculate the probability that a null hypothesis is true, in general?

I have a very simple question for Hypothesis testing. Is there a way to calculate the probability that the null hypothesis is false? Since in the literature they're always talking about the ...
153 views

### Is my interpretation of the p-value and confidence interval correct?

Background: let's say I have a website with a big buy button on it and I preformed an A/B testing with the following result: Version A (control): conversion rate 5% Version B (variation): ...
4k views

### FWER, FDR and multiple comparisons for Beginners

Wikipedia: "In statistics, family-wise error rate (FWER) is the probability of making one or more false discoveries, or type I errors, among all the hypotheses when performing multiple hypotheses ...