30 questions linked to/from Misunderstanding a P-value?
7k views

interpreting confidence intervals in t.test

I'm trying to run a t-test using t.test() but am confused about the p-value and confidence interval. Looking at the output I see a p-value of ~0.03. That leads me ...
466 views

Why is it necessary to give a-priori a threshhold for p-Values?

Let's say the results for an experiment give a p-Value of 0.0354234. Why is it necessary to fix a threshhold before doing the experiment and than report significance? Why do I not simply report the p-...
129 views

Do probabilities for selected type I error **rates** (alpha's) have confidence intervals?

In a journal article, there can be dozens of statistical tests that are unrelated, and, the more tests there are, the greater the likelihood that the probabilities for one of those tests for a given ...
6k views

Why is smaller the p-value, larger is the significance? [duplicate]

I am trying to understand the concept of p-value. Wikipedia mentions: The p-value is defined as the probability, under the assumption of hypothesis $H$, of obtaining a result equal to or more ...
21k views

Is Benjamini-Hochberg correction more conservative as the number of comparisons increases?

How conservative is Benjamini-Hochberg multiple testing correction relative to the total number of comparisons? For example, if I have a list of 18,000 features for two groups and I perform a Wilcoxon ...
235 views

Is the p-postulate (equal p-values provide equal evidence against the null) true?

The p-postulate is the notion that equal p-values provide equal evidence against the null hypothesis. Wagenmakers et al (2008) write: If p-values truly reflect evidence, a minimum requirement is ...
15k views

Can p-value be greater than 1?

Say I perform Student's t-test: can the p-value be greater than 1? The reason I ask this is that, some say that the p-value is the probability that we will have a Type I error and I would like to ...
2k views

What is an F test?

What is an F test and what does it show? Also what is an alpha, and how do I evaluate the p-value?
79 views

Am I taking crazy pills here or this A/B testing tool's documentation dead wrong?

In the documentation here for a popular A/B testing they say, For example, if you run an A/B testing experiment with a significance level of 95%, this means that if you determine a winner, you ...
195 views

Suppose I have a Cramér-von Mises $p = 0.99$ and Chi-squared $p = 0.88$ of a distribution being Student's-$t$. What can I say?

Suppose I have a Cramér-von Mises $p = 0.99$ and Chi-squared $p = 0.88$ of a distribution being Student's-$t$. What can I say? What can I say about the underlying distribution's being a Student's-$t$...
1k views

Can we calculate the probability that a null hypothesis is true, in general?

I have a very simple question for Hypothesis testing. Is there a way to calculate the probability that the null hypothesis is false? Since in the literature they're always talking about the ...
153 views

Is my interpretation of the p-value and confidence interval correct?

Background: let's say I have a website with a big buy button on it and I preformed an A/B testing with the following result: Version A (control): conversion rate 5% Version B (variation): ...
4k views

FWER, FDR and multiple comparisons for Beginners

Wikipedia: "In statistics, family-wise error rate (FWER) is the probability of making one or more false discoveries, or type I errors, among all the hypotheses when performing multiple hypotheses ...