Questions tagged [hypothesis-testing]
Hypothesis testing assesses whether data are inconsistent with a given hypothesis (usually a null hypothesis of no effect).
1,531 questions
382
votes
15
answers
153k
views
Is normality testing 'essentially useless'?
A former colleague once argued to me as follows:
We usually apply normality tests to the results of processes that,
under the null, generate random variables that are only
asymptotically or ...
- 15k
302
votes
15
answers
556k
views
What is the meaning of p values and t values in statistical tests?
After taking a statistics course and then trying to help fellow students, I noticed one subject that inspires much head-desk banging is interpreting the results of statistical hypothesis tests. It ...
- 4,454
75
votes
3
answers
63k
views
How to interpret type I, type II, and type III ANOVA and MANOVA?
My primary question is how to interpret the output (coefficients, F, P) when conducting a Type I (sequential) ANOVA?
My specific research problem is a bit more complex, so I will break my example ...
- 2,011
46
votes
3
answers
19k
views
Relation between confidence interval and testing statistical hypothesis for t-test
It is well known that confidence intervals and testing statistical hypothesis are strongly related. My questions is focused on comparison of means for two groups based on a numerical variable. Let's ...
- 1,449
145
votes
9
answers
123k
views
How to choose between t-test or non-parametric test e.g. Wilcoxon in small samples
Certain hypotheses can be tested using Student's t-test (maybe using Welch's correction for unequal variances in the two-sample case), or by a non-parametric test like the Wilcoxon paired signed rank ...
- 23.9k
62
votes
4
answers
21k
views
Why do statisticians say a non-significant result means "you can't reject the null" as opposed to accepting the null hypothesis?
Traditional statistical tests, like the two sample t-test, focus on trying to eliminate the hypothesis that there is no difference between a function of two independent samples. Then, we choose a ...
- 2,630
92
votes
10
answers
91k
views
Why is it possible to get significant F statistic (p<.001) but non-significant regressor t-tests?
In a multiple linear regression, why is it possible to have a highly significant F statistic (p<.001) but have very high p-values on all the regressor's t tests?
In my model, there are 10 ...
- 1,579
57
votes
5
answers
32k
views
Statistical inference when the sample "is" the population
Imagine you have to do reporting on the numbers of candidates who yearly take a given test. It seems rather difficult to infer the observed % of success, for instance, on a wider population due to the ...
- 1,261
186
votes
15
answers
56k
views
Are large data sets inappropriate for hypothesis testing?
In a recent article of Amstat News, the authors (Mark van der Laan and Sherri Rose) stated that "We know that for large enough sample sizes, every study—including ones in which the null hypothesis of ...
- 5,065
59
votes
1
answer
23k
views
Why do my p-values differ between logistic regression output, chi-squared test, and the confidence interval for the OR?
I have built a logistic regression where the outcome variable is being cured after receiving treatment (Cure vs. No Cure). All ...
- 790
20
votes
2
answers
10k
views
What are chunk tests?
In answer to a question on model selection in the presence of multicollinearity, Frank Harrell suggested:
Put all variables in the model but do not test for the effect of one
variable adjusted for ...
- 4,997
54
votes
3
answers
7k
views
Why does frequentist hypothesis testing become biased towards rejecting the null hypothesis with sufficiently large samples?
I was just reading this article on the Bayes factor for a completely unrelated problem when I stumbled upon this passage
Hypothesis testing with Bayes factors is more robust than frequentist ...
- 691
43
votes
6
answers
32k
views
Testing for autocorrelation: Ljung-Box versus Breusch-Godfrey
I am used to seeing Ljung-Box test used quite frequently for testing autocorrelation in raw data or in model residuals. I had nearly forgotten that there is another test for autocorrelation, namely, ...
- 69.5k
44
votes
3
answers
74k
views
Computing p-value using bootstrap with R
I use "boot" package to compute an approximated 2-sided bootstrapped p-value but the result is too far away from p-value of using t.test. I can't figure out what I did wrong in my R code. Can someone ...
- 2,977
16
votes
3
answers
11k
views
What follows if we fail to reject the null hypothesis? [duplicate]
What conclusions can we draw if $p>\alpha$? Does not rejecting the $H_0$ mean anything?
- 289
97
votes
5
answers
36k
views
When to use Fisher versus Neyman-Pearson framework?
I've been reading a lot lately about the differences between Fisher's method of hypothesis testing and the Neyman-Pearson school of thought.
My question is, ignoring philosophical objections, when ...
- 1,922
46
votes
4
answers
46k
views
Significance contradiction in linear regression: significant t-test for a coefficient vs non-significant overall F-statistic
I'm fitting a multiple linear regression model between 4 categorical variables (with 4 levels each) and a numerical output. My dataset has 43 observations.
Regression gives me the following $p$-...
- 2,634
34
votes
7
answers
14k
views
t-test for partially paired and partially unpaired data
An investigator wishes to produce a combined analysis of several datasets. In some datasets there are paired observations for treatment A and B. In others there are unpaired A and/or B data. I am ...
- 98.4k
119
votes
7
answers
214k
views
T-test for non normal when N>50?
Long ago I learnt that normal distribution was necessary to use a two sample T-test. Today a colleague told me that she learnt that for N>50 normal distribution was not necessary. Is that true?
If ...
- 2,397
100
votes
11
answers
280k
views
How to obtain the p-value (check significance) of an effect in a lme4 mixed model?
I use lme4 in R to fit the mixed model
lmer(value~status+(1|experiment)))
where value is continuous, status and experiment are factors, and I get
...
- 2,201
30
votes
3
answers
34k
views
How can I calculate margin of error in a NPS (Net Promoter Score) result?
I'll let Wikipedia explain how NPS is calculated:
The Net Promoter Score is obtained by asking customers a single
question on a 0 to 10 rating scale, where 10 is "extremely likely" and
0 is "...
- 401
19
votes
1
answer
69k
views
How do I find values not given in (interpolate in) statistical tables?
Often people use programs to obtain p-values, but sometimes - for whatever reason - it may be necessary to obtain a critical value from a set of tables.
Given a statistical table with a limited ...
- 290k
30
votes
2
answers
27k
views
The order of variables in ANOVA matters, doesn't it?
Am I correct to understand that the order in which variables are specified in a multifactorial ANOVA makes a difference but that the order does not matter when doing a multiple linear regression?
So ...
- 750
24
votes
4
answers
11k
views
Misunderstanding a P-value?
So I've been reading a lot about how to correctly interpret a P-value, and from what I've read, the p-value says NOTHING about the probability that the null hypothesis is true or false. However, when ...
- 726
76
votes
4
answers
140k
views
Testing equality of coefficients from two different regressions
This seems to be a basic issue, but I just realized that I actually don't know how to test equality of coefficients from two different regressions. Can anyone shed some light on this?
More formally, ...
- 2,046
41
votes
8
answers
26k
views
How to test hypothesis of no group differences?
Imagine you have a study with two groups (e.g., males and females) looking at a numeric dependent variable (e.g., intelligence test scores) and you have the hypothesis that there are no group ...
- 45.7k
17
votes
5
answers
9k
views
A/B testing ratio of sums
Context
Consider the following scenario for a company selling goods online. A user can purchase several items (i.e. basket of items), some of which are of particular importance and are tracked ...
- 9,976
110
votes
3
answers
22k
views
What are examples where a "naive bootstrap" fails?
Suppose I have a set of sample data from an unknown or complex distribution, and I want to perform some inference on a statistic $T$ of the data. My default inclination is to just generate a bunch of ...
- 10.1k
45
votes
4
answers
10k
views
Interpretation of p-value in hypothesis testing
I recently came across the paper "The Insignificance of Null Hypothesis Significance Testing", Jeff Gill (1999). The author raised a few common misconceptions regarding hypothesis testing and p-values,...
user13587
27
votes
4
answers
19k
views
Comparing and contrasting, p-values, significance levels and type I error
I was wondering if anybody could give a concise rundown as to the definitions and uses of p-values, significance level and type I error.
I understand that p-values are defined as "the probability of ...
- 1,505
12
votes
2
answers
4k
views
Can we reject a null hypothesis with confidence intervals produced via sampling rather than the null hypothesis?
I have been taught that we can produce a parameter estimate in the form of a confidence interval after sampling from a population. For example, 95% confidence intervals, with no violated assumptions, ...
- 321
37
votes
4
answers
42k
views
Checking if two Poisson samples have the same mean
This is an elementary question, but I wasn't able to find the answer. I have two measurements: n1 events in time t1 and n2 events in time t2, both produced (say) by Poisson processes with possibly-...
- 1,248
34
votes
5
answers
16k
views
Prove that F-statistic follows F-distribution
In light of this question : Proof that the coefficients in an OLS model follow a t-distribution with (n-k) degrees of freedom
I would love to understand why
$$ F = \frac{(\text{TSS}-\text{RSS})/(p-1)...
- 443
31
votes
2
answers
20k
views
Choosing between $z$-test and $t$-test
Background: I'm giving a presentation to colleagues at work on hypothesis testing, and understand most of it fine but there's one aspect that I'm tying myself up in knots trying to understand as well ...
- 313
27
votes
1
answer
39k
views
Why Levene test of equality of variances rather than F ratio?
SPSS uses the Levene test to evaluate homogeneity of variances in the independent group t-test procedure.
Why is the Levene test better than a simple F ratio of the ratio of the variances of the ...
- 3,427
24
votes
5
answers
13k
views
What exactly does a non-parametric test accomplish & What do you do with the results?
I have a feeling this may have been asked elsewhere, but not really with the type of basic description I need. I know non-parametric relies on the median instead of the mean to compare... something. ...
- 315
100
votes
2
answers
14k
views
How much do we know about p-hacking "in the wild"?
The phrase p-hacking (also: "data dredging", "snooping" or "fishing") refers to various kinds of statistical malpractice in which results become artificially statistically significant. There are many ...
- 23.9k
39
votes
2
answers
5k
views
Understanding p-value
I know that there are lots of materials explaining p-value. However the concept is not easy to grasp firmly without further clarification.
Here is the definition of p-value from Wikipedia:
The p-...
- 501
23
votes
2
answers
20k
views
What is the difference between "testing of hypothesis" and "test of significance"?
Is there a difference between the phrases "testing of hypothesis" and "test of significance" or are they the same?
After a detailed answer from @Micheal Lew, I have one confusion that nowadays ...
- 1,284
23
votes
1
answer
3k
views
Is there a statistical test to compare two samples of size 1 and 3?
For an ecology project, my lab group added vinegar to 4 tanks containing equal volumes of pond water, 1 control with no elodea (an aquatic plant) and 3 treatments with the same amount of elodea in ...
- 333
20
votes
3
answers
23k
views
What tests do I use to confirm that residuals are normally distributed?
I have some data which looks from plotting a graph of residuals vs time almost normal but I want to be sure. How can I test for normality of error residuals?
- 201
77
votes
15
answers
13k
views
Why would parametric statistics ever be preferred over nonparametric?
Can someone explain to me why would anyone choose a parametric over a nonparametric statistical method for hypothesis testing or regression analysis?
In my mind, it's like going for rafting and ...
- 957
71
votes
7
answers
148k
views
Is chi-squared always a one-sided test?
A published article (pdf) contains these 2 sentences:
Moreover, misreporting may be caused by the application of incorrect rules or by a lack of knowledge of the statistical test. For example, the ...
- 3,427
46
votes
4
answers
8k
views
Do null and alternative hypotheses have to be exhaustive or not?
I saw a lot of times claims that they have to be exhaustive (the examples in such books were always set in such way, that they were indeed), on the other hand I also saw a lot of times books stating ...
- 683
14
votes
2
answers
2k
views
Family-wise error boundary: Does re-using data sets on different studies of independent questions lead to multiple testing problems?
If a team of researchers perform multiple (hypothesis) tests on a given data set, there is a volume of literature asserting that they should use some form of correction for multiple testing (...
- 243
87
votes
10
answers
25k
views
Regarding p-values, why 1% and 5%? Why not 6% or 10%?
Regarding p-values, I am wondering why $1$% and $5$% seem to be the gold standard for "statistical significance". Why not other values, like $6$% or $10$%?
Is ...
- 1,499
40
votes
8
answers
25k
views
Is it possible to prove a null hypothesis?
As the question states - Is it possible to prove the null hypothesis? From my (limited) understanding of hypothesis, the answer is no but I can't come up with a rigorous explanation for it. Does the ...
- 511
33
votes
2
answers
32k
views
Wald test in regression (OLS and GLMs): t- vs. z-distribution
I understand that the Wald test for regression coefficients is based on the following property that holds asymptotically (e.g. Wasserman (2006): All of Statistics, pages 153, 214-215):
$$
\frac{(\hat{...
- 31.2k
16
votes
5
answers
2k
views
Does p-value ever depend on the alternative?
Our tag definition of the $p$-value says
In frequentist hypothesis testing, the $p$-value is the probability of a result as extreme (or more) than the observed result, under the assumption that the ...
- 69.5k
117
votes
10
answers
8k
views
ASA discusses limitations of $p$-values - what are the alternatives?
We already have multiple threads tagged as p-values that reveal lots of misunderstandings about them. Ten months ago we had a thread about psychological journal that "banned" $p$-values, now American ...
- 141k