Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [hypothesis-testing]

Hypothesis testing assesses whether data are inconsistent with a given hypothesis rather than being an effect of random fluctuations.

276
votes
16answers
90k views

Is normality testing 'essentially useless'?

A former colleague once argued to me as follows: We usually apply normality tests to the results of processes that, under the null, generate random variables that are only asymptotically or ...
228
votes
14answers
437k views

What is the meaning of p values and t values in statistical tests?

After taking a statistics course and then trying to help fellow students, I noticed one subject that inspires much head-desk banging is interpreting the results of statistical hypothesis tests. It ...
86
votes
7answers
70k views

How to choose between t-test or non-parametric test e.g. Wilcoxon in small samples

Certain hypotheses can be tested using Student's t-test (maybe using Welch's correction for unequal variances in the two-sample case), or by a non-parametric test like the Wilcoxon paired signed rank ...
41
votes
4answers
12k views

Why do statisticians say a non-significant result means “you can't reject the null” as opposed to accepting the null hypothesis?

Traditional statistical tests, like the two sample t-test, focus on trying to eliminate the hypothesis that there is no difference between a function of two independent samples. Then, we choose a ...
38
votes
1answer
29k views

How to interpret type I, type II, and type III ANOVA and MANOVA?

My primary question is how to interpret the output (coefficients, F, P) when conducting a Type I (sequential) ANOVA? My specific research problem is a bit more complex, so I will break my example ...
41
votes
5answers
22k views

Statistical inference when the sample “is” the population

Imagine you have to do reporting on the numbers of candidates who yearly take a given test. It seems rather difficult to infer the observed % of success, for instance, on a wider population due to the ...
27
votes
3answers
10k views

Relation between confidence interval and testing statistical hypothesis for t-test

It is well known that confidence intervals and testing statistical hypothesis are strongly related. My questions is focused on comparison of means for two groups based on a numerical variable. Let's ...
124
votes
14answers
31k views

Are large data sets inappropriate for hypothesis testing?

In a recent article of Amstat News, the authors (Mark van der Laan and Sherri Rose) stated that "We know that for large enough sample sizes, every study—including ones in which the null hypothesis of ...
13
votes
3answers
3k views

What follows if we fail to reject the null hypothesis? [duplicate]

What conclusions can we draw if $p>\alpha$? Does not rejecting the $H_0$ mean anything?
65
votes
9answers
39k views

Why is it possible to get significant F statistic (p<.001) but non-significant regressor t-tests?

In a multiple linear regression, why is it possible to have a highly significant F statistic (p<.001) but have very high p-values on all the regressor's t tests? In my model, there are 10 ...
33
votes
1answer
7k views

Why do my p-values differ between logistic regression output, chi-squared test, and the confidence interval for the OR?

I have built a logistic regression where the outcome variable is being cured after receiving treatment (Cure vs. No Cure). All ...
43
votes
2answers
4k views

Why does frequentist hypothesis testing become biased towards rejecting the null hypothesis with sufficiently large samples?

I was just reading this article on the Bayes factor for a completely unrelated problem when I stumbled upon this passage Hypothesis testing with Bayes factors is more robust than frequentist ...
19
votes
1answer
44k views

How do I find values not given in (interpolate in) statistical tables?

Often people use programs to obtain p-values, but sometimes - for whatever reason - it may be necessary to obtain a critical value from a set of tables. Given a statistical table with a limited ...
17
votes
4answers
5k views

Misunderstanding a P-value?

So I've been reading a lot about how to correctly interpret a P-value, and from what I've read, the p-value says NOTHING about the probability that the null hypothesis is true or false. However, when ...
65
votes
4answers
24k views

When to use Fisher and Neyman-Pearson framework?

I've been reading a lot lately about the differences between Fisher's method of hypothesis testing and the Neyman-Pearson school of thought. My question is, ignoring philosophical objections for a ...
34
votes
3answers
36k views

Significance contradiction in linear regression: significant t-test for a coefficient vs non-significant overall F-statistic

I'm fitting a multiple linear regression model between 4 categorical variables (with 4 levels each) and a numerical output. My dataset has 43 observations. Regression gives me the following $p$-...
37
votes
8answers
18k views

How to test hypothesis of no group differences?

Imagine you have a study with two groups (e.g., males and females) looking at a numeric dependent variable (e.g., intelligence test scores) and you have the hypothesis that there are no group ...
18
votes
1answer
26k views

Why Levene test of equality of variances rather than F ratio?

SPSS uses the Levene test to evaluate homogeneity of variances in the independent group t-test procedure. Why is the Levene test better than a simple F ratio of the ratio of the variances of the ...
22
votes
5answers
9k views

What exactly does a non-parametric test accomplish & What do you do with the results?

I have a feeling this may have been asked elsewhere, but not really with the type of basic description I need. I know non-parametric relies on the median instead of the mean to compare... something. ...
20
votes
3answers
20k views

How can I calculate margin of error in a NPS (Net Promoter Score) result?

I'll let Wikipedia explain how NPS is calculated: The Net Promoter Score is obtained by asking customers a single question on a 0 to 10 rating scale, where 10 is "extremely likely" and 0 is "...
18
votes
3answers
17k views

What tests do I use to confirm that residuals are normally distributed?

I have some data which looks from plotting a graph of residuals vs time almost normal but I want to be sure. How can I test for normality of error residuals?
46
votes
8answers
98k views

How to obtain the p-value (check significance) of an effect in a lme4 mixed model?

I use lme4 in R to fit the mixed model lmer(value~status+(1|experiment))) where value is continuous, status and experiment are factors, and I get ...
33
votes
2answers
3k views

Understanding p-value

I know that there are lots of materials explaining p-value. However the concept is not easy to grasp firmly without further clarification. Here is the definition of p-value from Wikipedia: The p-...
19
votes
3answers
16k views

Comparing and contrasting, p-values, significance levels and type I error

I was wondering if anybody could give a concise rundown as to the definitions and uses of p-values, significance level and type I error. I understand that p-values are defined as "the probability of ...
27
votes
5answers
15k views

Testing for autocorrelation: Ljung-Box versus Breusch-Godfrey

I am used to seeing Ljung-Box test used quite frequently for testing autocorrelation in raw data or in model residuals. I had nearly forgotten that there is another test for autocorrelation, namely, ...
19
votes
1answer
2k views

Is there a statistical test to compare two samples of size 1 and 3?

For an ecology project, my lab group added vinegar to 4 tanks containing equal volumes of pond water, 1 control with no elodea (an aquatic plant) and 3 treatments with the same amount of elodea in ...
82
votes
3answers
11k views

What are examples where a “naive bootstrap” fails?

Suppose I have a set of sample data from an unknown or complex distribution, and I want to perform some inference on a statistic $T$ of the data. My default inclination is to just generate a bunch of ...
25
votes
5answers
6k views

t-test for partially paired and partially unpaired data

An investigator wishes to produce a combined analysis of several datasets. In some datasets there are paired observations for treatment A and B. In others there are unpaired A and/or B data. I am ...
11
votes
2answers
816 views

Family-wise error boundary: Does re-using data sets on different studies of independent questions lead to multiple testing problems?

If a team of researchers perform multiple (hypothesis) tests on a given data set, there is a volume of literature asserting that they should use some form of correction for multiple testing (...
34
votes
3answers
7k views

Interpretation of p-value in hypothesis testing

I recently came across the paper "The Insignificance of Null Hypothesis Significance Testing", Jeff Gill (1999). The author raised a few common misconceptions regarding hypothesis testing and p-values,...
46
votes
8answers
88k views

Is chi-squared always a one-sided test?

A published article (pdf) contains these 2 sentences: Moreover, misreporting may be caused by the application of incorrect rules or by a lack of knowledge of the statistical test. For example, the ...
2
votes
1answer
7k views

Testing for moderation with continuous vs. categorical moderators

I am testing an interaction effect where $X$ and $Y$ are continuous variable and $M$ (Moderator) is a categorical variable (effects coding $+1$, $-1$). I have no clue about how to do a post-hoc ...
36
votes
8answers
16k views

Is it possible to prove a null hypothesis?

As the question states - Is it possible to prove the null hypothesis? From my (limited) understanding of hypothesis, the answer is no but I can't come up with a rigorous explanation for it. Does the ...
78
votes
9answers
15k views

Regarding p-values, why 1% and 5%? Why not 6% or 10%?

Regarding p-values, I am wondering why $1$% and $5$% seem to be the gold standard for "statistical significance". Why not other values, like $6$% or $10$%? Is ...
25
votes
3answers
25k views

Computing p-value using bootstrap with R

I use "boot" package to compute an approximated 2-sided bootstrapped p-value but the result is too far away from p-value of using t.test. I can't figure out what I did wrong in my R code. Can someone ...
25
votes
4answers
24k views

Checking if two Poisson samples have the same mean

This is an elementary question, but I wasn't able to find the answer. I have two measurements: n1 events in time t1 and n2 events in time t2, both produced (say) by Poisson processes with possibly-...
13
votes
8answers
2k views

Testing random variate generation algorithms

Which methods are used for testing random variate generation algorithms?
59
votes
3answers
18k views

Explain the xkcd jelly bean comic: What makes it funny?

I see that one time out of the twenty total tests they run, $p < 0.05$, so they wrongly assume that during one of the twenty tests, the result is significant ($0.05 = 1/20$). xkcd jelly bean ...
26
votes
3answers
62k views

What is the difference between confidence intervals and hypothesis testing?

I have read about controversies regarding hypothesis testing with some commentators suggesting that hypothesis testing should not be used. Some commentators suggest that confidence intervals should ...
26
votes
2answers
2k views

Is rejecting the hypothesis using p-value equivalent to hypothesis not belonging to the confidence interval?

While formally deriving the confidence interval of an estimate, I ended up with a formula that resembles very closely the way $p$-value is computed. Thus the question: are they formally equivalent? I....
24
votes
2answers
3k views

Do null and alternative hypotheses have to be exhaustive or not?

I saw a lot of times claims that they have to be exhaustive (the examples in such books were always set in such way, that they were indeed), on the other hand I also saw a lot of times books stating ...
11
votes
1answer
44k views

Post-hoc tests after Kruskal-Wallis: Dunn's test or Bonferroni corrected Mann-Whitney tests?

I have some non-Gaussian distributed variable and I need to check if there are significant differences between the values of this variable in 5 different groups. I have performed Kruskal-Wallis one-...
12
votes
1answer
4k views

Goodness-of-fit test in Logistic regression; which 'fit' do we want to test?

I am referring to the question and its answers: How to compare (probability) predictive ability of models developed from logistic regression? by @Clark Chong and answers/comments by @Frank Harrell. ...
87
votes
7answers
4k views

ASA discusses limitations of $p$-values - what are the alternatives?

We already have multiple threads tagged as p-values that reveal lots of misunderstandings about them. Ten months ago we had a thread about psychological journal that "banned" $p$-values, now American ...
94
votes
2answers
9k views

How much do we know about p-hacking “in the wild”?

The phrase p-hacking (also: "data dredging", "snooping" or "fishing") refers to various kinds of statistical malpractice in which results become artificially statistically significant. There are many ...
19
votes
2answers
13k views

Choosing between $z$-test and $t$-test

Background: I'm giving a presentation to colleagues at work on hypothesis testing, and understand most of it fine but there's one aspect that I'm tying myself up in knots trying to understand as well ...
34
votes
2answers
50k views

Testing equality of coefficients from two different regressions

This seems to be a basic issue, but I just realized that I actually don't know how to test equality of coefficients from two different regressions. Can anyone shed some light on this? More formally, ...
35
votes
6answers
2k views

Effect size as the hypothesis for significance testing

Today, at the Cross Validated Journal Club (why weren't you there?), @mbq asked: Do you think we (modern data scientists) know what significance means? And how it relates to our confidence in our ...
16
votes
2answers
40k views

What is the correct way to test for significant differences between coefficients?

I'm hoping someone can help straighten out a point of confusion for me. Say I want to test whether 2 sets of regression coefficients are significantly different from each other, with the following set ...
18
votes
2answers
19k views

Wald test in regression (OLS and GLMs): t- vs. z-distribution

I understand that the Wald test for regression coefficients is based on the following property that holds asymptotically (e.g. Wasserman (2006): All of Statistics, pages 153, 214-215): $$ \frac{(\hat{...