Linked Questions

246
votes
16answers
463k views

What is the meaning of p values and t values in statistical tests?

After taking a statistics course and then trying to help fellow students, I noticed one subject that inspires much head-desk banging is interpreting the results of statistical hypothesis tests. It ...
98
votes
9answers
14k views

Is this really how p-values work? Can a million research papers per year be based on pure randomness?

I'm very new to statistics, and I'm just learning to understand the basics, including $p$-values. But there is a huge question mark in my mind right now, and I kind of hope my understanding is wrong. ...
36
votes
3answers
7k views

Interpretation of p-value in hypothesis testing

I recently came across the paper "The Insignificance of Null Hypothesis Significance Testing", Jeff Gill (1999). The author raised a few common misconceptions regarding hypothesis testing and p-values,...
20
votes
8answers
9k views

Stats is not maths?

Is stats maths or not? Given that it's all numbers, mostly taught by maths departments and you get maths credits for it, I wonder whether people just mean it half-jokingly when they say it, like ...
21
votes
3answers
17k views

Comparing and contrasting, p-values, significance levels and type I error

I was wondering if anybody could give a concise rundown as to the definitions and uses of p-values, significance level and type I error. I understand that p-values are defined as "the probability of ...
31
votes
4answers
9k views

Are smaller p-values more convincing?

I've been reading up on $p$-values, type 1 error rates, significance levels, power calculations, effect sizes and the Fisher vs Neyman-Pearson debate. This has left me feeling a bit overwhelmed. I ...
19
votes
1answer
21k views

Is Benjamini-Hochberg correction more conservative as the number of comparisons increases?

How conservative is Benjamini-Hochberg multiple testing correction relative to the total number of comparisons? For example, if I have a list of 18,000 features for two groups and I perform a Wilcoxon ...
14
votes
3answers
4k views

What follows if we fail to reject the null hypothesis? [duplicate]

What conclusions can we draw if $p>\alpha$? Does not rejecting the $H_0$ mean anything?
9
votes
6answers
2k views

Using p-value to compute the probability of hypothesis being true; what else is needed?

Question: One common misunderstanding of p-values is that they represent the probability of the null hypothesis being true. I know that's not correct and I know that p-values only represent the ...
14
votes
4answers
4k views

Probability that Null Hypothesis is True

So, this may be a common question, but I’ve never found a satisfactory answer. How do you determine the probability that the null hypothesis is true (or false)? Let’s say you give students two ...
8
votes
5answers
2k views

What is the relationship between $p$ values and Type I errors [duplicate]

In hypothesis testing we set an accepted level of Type I error probability $\alpha$ and observe whether a sample statistic is equally likely or less likely to be observed if the null hypothesis was ...
5
votes
1answer
15k views

Can p-value be greater than 1?

Say I perform Student's t-test: can the p-value be greater than 1? The reason I ask this is that, some say that the p-value is the probability that we will have a Type I error and I would like to ...
33
votes
2answers
3k views

Degrees of freedom of $\chi^2$ in Hosmer-Lemeshow test

The test statistic for the Hosmer-Lemeshow test (HLT) for goodness of fit (GOF) of a logistic regression model is defined as follows: The sample is then split into $d=10$ deciles, $D_1, D_2, \dots ...
3
votes
3answers
9k views

Chi-Square-Test: Why is the chi-squared test a one-tailed test? [duplicate]

In my Masters thesis I conducted an experiment in that subjects should decide between two alternatives. I had a control and a treatment group. The hypothesis was that in the treatment group more ...
11
votes
2answers
971 views

Family-wise error boundary: Does re-using data sets on different studies of independent questions lead to multiple testing problems?

If a team of researchers perform multiple (hypothesis) tests on a given data set, there is a volume of literature asserting that they should use some form of correction for multiple testing (...
2
votes
2answers
7k views

interpreting confidence intervals in t.test

I'm trying to run a t-test using t.test() but am confused about the p-value and confidence interval. Looking at the output I see a p-value of ~0.03. That leads me ...
5
votes
1answer
2k views

What is an F test?

What is an F test and what does it show? Also what is an alpha, and how do I evaluate the p-value?
4
votes
3answers
2k views

Why do we use the chi-squared test to test independence of 2 categorical variables?

I sometimes use the chi-squared test to test the independence of categorical variables. I know how to do the test but don't know the underlying concepts and details. So, I was wondering about the ...
0
votes
2answers
6k views

Why is smaller the p-value, larger is the significance? [duplicate]

I am trying to understand the concept of p-value. Wikipedia mentions: The p-value is defined as the probability, under the assumption of hypothesis $H$, of obtaining a result equal to or more ...
2
votes
1answer
4k views

FWER, FDR and multiple comparisons for Beginners

Wikipedia: "In statistics, family-wise error rate (FWER) is the probability of making one or more false discoveries, or type I errors, among all the hypotheses when performing multiple hypotheses ...
3
votes
1answer
1k views

Can we calculate the probability that a null hypothesis is true, in general?

I have a very simple question for Hypothesis testing. Is there a way to calculate the probability that the null hypothesis is false? Since in the literature they're always talking about the ...
1
vote
2answers
474 views

Why is it necessary to give a-priori a threshhold for p-Values?

Let's say the results for an experiment give a p-Value of 0.0354234. Why is it necessary to fix a threshhold before doing the experiment and than report significance? Why do I not simply report the p-...
3
votes
1answer
196 views

Suppose I have a Cramér-von Mises $p = 0.99$ and Chi-squared $p = 0.88$ of a distribution being Student's-$t$. What can I say?

Suppose I have a Cramér-von Mises $p = 0.99$ and Chi-squared $p = 0.88$ of a distribution being Student's-$t$. What can I say? What can I say about the underlying distribution's being a Student's-$t$...
0
votes
3answers
96 views

Why do we do hypothesis testing on estimates of linear regression?

I was reading about linear regression and what I understood is that once we minimize OLS equation we get the beta parameters. Its just like solving a normal equation to get the unknowns. Then why do ...
1
vote
1answer
415 views

Confusion regarding p-values and false discovery rate

I get stuck with understanding of the following two statements (from Wikipedia on p-values): The p-value is the probability of obtaining at least as extreme results given that the null hypothesis ...
6
votes
1answer
235 views

Is the p-postulate (equal p-values provide equal evidence against the null) true?

The p-postulate is the notion that equal p-values provide equal evidence against the null hypothesis. Wagenmakers et al (2008) write: If p-values truly reflect evidence, a minimum requirement is ...
4
votes
1answer
79 views

Am I taking crazy pills here or this A/B testing tool's documentation dead wrong?

In the documentation here for a popular A/B testing they say, For example, if you run an A/B testing experiment with a significance level of 95%, this means that if you determine a winner, you ...
2
votes
1answer
153 views

Is my interpretation of the p-value and confidence interval correct?

Background: let's say I have a website with a big buy button on it and I preformed an A/B testing with the following result: Version A (control): conversion rate 5% Version B (variation): ...
0
votes
2answers
129 views

Do probabilities for selected type I error **rates** (alpha's) have confidence intervals?

In a journal article, there can be dozens of statistical tests that are unrelated, and, the more tests there are, the greater the likelihood that the probabilities for one of those tests for a given ...
1
vote
1answer
51 views

Interpretation of the D-statistic from 2-sample KS-tests

I've been using the two sample Kolmogorov-Smirnov test to compare observations from 2018 vs observations from 2019. I heard from my co-worker that you can interpret the D-statistic as the magnitude ...