Questions tagged [hypothesis-testing]
Hypothesis testing assesses whether data are inconsistent with a given hypothesis (usually a null hypothesis of no effect).
10,986 questions
382
votes
15
answers
153k
views
Is normality testing 'essentially useless'?
A former colleague once argued to me as follows:
We usually apply normality tests to the results of processes that,
under the null, generate random variables that are only
asymptotically or ...
- 15k
302
votes
15
answers
556k
views
What is the meaning of p values and t values in statistical tests?
After taking a statistics course and then trying to help fellow students, I noticed one subject that inspires much head-desk banging is interpreting the results of statistical hypothesis tests. It ...
- 4,454
186
votes
15
answers
56k
views
Are large data sets inappropriate for hypothesis testing?
In a recent article of Amstat News, the authors (Mark van der Laan and Sherri Rose) stated that "We know that for large enough sample sizes, every study—including ones in which the null hypothesis of ...
- 5,065
145
votes
9
answers
123k
views
How to choose between t-test or non-parametric test e.g. Wilcoxon in small samples
Certain hypotheses can be tested using Student's t-test (maybe using Welch's correction for unequal variances in the two-sample case), or by a non-parametric test like the Wilcoxon paired signed rank ...
- 23.9k
142
votes
9
answers
29k
views
Is Facebook coming to an end?
Recently, this paper has received a lot of attention (e.g. from WSJ). Basically, the authors conclude that Facebook will lose 80% of its members by 2017.
They base their claims on an extrapolation ...
- 1,033
119
votes
7
answers
214k
views
T-test for non normal when N>50?
Long ago I learnt that normal distribution was necessary to use a two sample T-test. Today a colleague told me that she learnt that for N>50 normal distribution was not necessary. Is that true?
If ...
- 2,397
117
votes
10
answers
8k
views
ASA discusses limitations of $p$-values - what are the alternatives?
We already have multiple threads tagged as p-values that reveal lots of misunderstandings about them. Ten months ago we had a thread about psychological journal that "banned" $p$-values, now American ...
- 141k
110
votes
3
answers
22k
views
What are examples where a "naive bootstrap" fails?
Suppose I have a set of sample data from an unknown or complex distribution, and I want to perform some inference on a statistic $T$ of the data. My default inclination is to just generate a bunch of ...
- 10.1k
105
votes
9
answers
15k
views
Is this really how p-values work? Can a million research papers per year be based on pure randomness?
I'm very new to statistics, and I'm just learning to understand the basics, including $p$-values. But there is a huge question mark in my mind right now, and I kind of hope my understanding is wrong. ...
- 1,111
100
votes
11
answers
280k
views
How to obtain the p-value (check significance) of an effect in a lme4 mixed model?
I use lme4 in R to fit the mixed model
lmer(value~status+(1|experiment)))
where value is continuous, status and experiment are factors, and I get
...
- 2,201
100
votes
2
answers
14k
views
How much do we know about p-hacking "in the wild"?
The phrase p-hacking (also: "data dredging", "snooping" or "fishing") refers to various kinds of statistical malpractice in which results become artificially statistically significant. There are many ...
- 23.9k
97
votes
5
answers
36k
views
When to use Fisher versus Neyman-Pearson framework?
I've been reading a lot lately about the differences between Fisher's method of hypothesis testing and the Neyman-Pearson school of thought.
My question is, ignoring philosophical objections, when ...
- 1,922
92
votes
10
answers
91k
views
Why is it possible to get significant F statistic (p<.001) but non-significant regressor t-tests?
In a multiple linear regression, why is it possible to have a highly significant F statistic (p<.001) but have very high p-values on all the regressor's t tests?
In my model, there are 10 ...
- 1,579
87
votes
10
answers
25k
views
Regarding p-values, why 1% and 5%? Why not 6% or 10%?
Regarding p-values, I am wondering why $1$% and $5$% seem to be the gold standard for "statistical significance". Why not other values, like $6$% or $10$%?
Is ...
- 1,499
77
votes
15
answers
13k
views
Why would parametric statistics ever be preferred over nonparametric?
Can someone explain to me why would anyone choose a parametric over a nonparametric statistical method for hypothesis testing or regression analysis?
In my mind, it's like going for rafting and ...
- 957
76
votes
4
answers
140k
views
Testing equality of coefficients from two different regressions
This seems to be a basic issue, but I just realized that I actually don't know how to test equality of coefficients from two different regressions. Can anyone shed some light on this?
More formally, ...
- 2,046
75
votes
3
answers
63k
views
How to interpret type I, type II, and type III ANOVA and MANOVA?
My primary question is how to interpret the output (coefficients, F, P) when conducting a Type I (sequential) ANOVA?
My specific research problem is a bit more complex, so I will break my example ...
- 2,011
75
votes
4
answers
6k
views
A psychology journal banned p-values and confidence intervals; is it indeed wise to stop using them?
On 25 February 2015, the journal Basic and Applied Social Psychology issued an editorial banning $p$-values and confidence intervals from all future papers.
Specifically, they say (formatting and ...
73
votes
6
answers
5k
views
Is the "hybrid" between Fisher and Neyman-Pearson approaches to statistical testing really an "incoherent mishmash"?
There exists a certain school of thought according to which the most widespread approach to statistical testing is a "hybrid" between two approaches: that of Fisher and that of Neyman-...
- 107k
73
votes
3
answers
4k
views
Is this the solution to the p-value problem?
In February 2016, the American Statistical Association released a formal statement on statistical significance and p-values. Our thread about it discusses these issues extensively. However, no ...
71
votes
7
answers
148k
views
Is chi-squared always a one-sided test?
A published article (pdf) contains these 2 sentences:
Moreover, misreporting may be caused by the application of incorrect rules or by a lack of knowledge of the statistical test. For example, the ...
- 3,427
70
votes
5
answers
139k
views
What is the difference between a "nested" and a "non-nested" model?
In the literature on hierarchical/multilevel models I have often read about "nested models" and "non-nested models", but what does this mean? Could anyone maybe give me some examples or tell me about ...
- 851
69
votes
8
answers
8k
views
What is a good, convincing example in which p-values are useful?
My question in the title is self explanatory, but I would like to give it some context.
The ASA released a statement earlier this week “on p-values: context, process, and purpose”, outlining various ...
- 21.9k
69
votes
3
answers
6k
views
References containing arguments against null hypothesis significance testing?
In the last few years I've read a number of papers arguing against the use of null hypothesis significance testing in science, but didn't think to keep a persistent list. A colleague recently asked me ...
68
votes
5
answers
6k
views
Why does collecting data until finding a significant result increase Type I error rate?
I was wondering exactly why collecting data until a significant result (e.g., $p \lt .05$) is obtained (i.e., p-hacking) increases the Type I error rate?
I would also highly appreciate an ...
- 976
67
votes
14
answers
23k
views
If we fail to reject the null hypothesis in a large study, isn't it evidence for the null?
A basic limitation of null hypothesis significance testing is that it does not allow a researcher to gather evidence in favor of the null (Source)
I see this claim repeated in multiple places, but I ...
- 1,309
67
votes
7
answers
52k
views
How do I test that two continuous variables are independent?
Suppose I have a sample $(X_n,Y_n), n=1..N$ from the joint distribution of $X$ and $Y$. How do I test the hypothesis that $X$ and $Y$ are independent?
No assumption is made on the joint or marginal ...
- 2,256
66
votes
1
answer
8k
views
40,000 neuroscience papers might be wrong
I saw this article in the Economist about a seemingly devastating paper [1] casting doubt on "something like 40,000 published [fMRI] studies." The error, they say, is because of "erroneous statistical ...
- 2,312
65
votes
3
answers
30k
views
Explain the xkcd jelly bean comic: What makes it funny?
I see that one time out of the twenty total tests they run, $p < 0.05$, so they wrongly assume that during one of the twenty tests, the result is significant ($0.05 = 1/20$).
xkcd jelly bean ...
- 713
64
votes
13
answers
11k
views
Two-tailed tests... I'm just not convinced. What's the point?
The following excerpt is from the entry, What are the differences between one-tailed and two-tailed tests?, on UCLA's statistics help site.
... consider the consequences of missing an effect in the ...
- 773
62
votes
4
answers
21k
views
Why do statisticians say a non-significant result means "you can't reject the null" as opposed to accepting the null hypothesis?
Traditional statistical tests, like the two sample t-test, focus on trying to eliminate the hypothesis that there is no difference between a function of two independent samples. Then, we choose a ...
- 2,630
60
votes
3
answers
30k
views
When combining p-values, why not just averaging?
I recently learned about Fisher's method to combine p-values. This is based on the fact that p-value under the null follows a uniform distribution, and that $$-2\sum_{i=1}^n{\log X_i} \sim \chi^2(2n), ...
- 2,253
59
votes
1
answer
23k
views
Why do my p-values differ between logistic regression output, chi-squared test, and the confidence interval for the OR?
I have built a logistic regression where the outcome variable is being cured after receiving treatment (Cure vs. No Cure). All ...
- 790
58
votes
8
answers
7k
views
Why continue to teach and use hypothesis testing (when confidence intervals are available)?
Why continue to teach and use hypothesis testing (with all its difficult concepts and which are among the most statistical sins) for problems where there is an interval estimator (confidence, ...
57
votes
5
answers
32k
views
Statistical inference when the sample "is" the population
Imagine you have to do reporting on the numbers of candidates who yearly take a given test. It seems rather difficult to infer the observed % of success, for instance, on a wider population due to the ...
- 1,261
56
votes
5
answers
45k
views
Correct spelling (capitalization, italicization, hyphenation) of "p-value"?
I realize this is pedantic and trite, but as a researcher in a field outside of statistics, with limited formal education in statistics, I always wonder if I'm writing "p-value" correctly. ...
- 953
55
votes
5
answers
27k
views
Bayesian equivalent of two sample t-test?
I'm not looking for a plug and play method like BEST in R but rather a mathematical explanation of what are some Bayesian methods I can use to test the difference between the mean of two samples.
- 581
54
votes
3
answers
7k
views
Why does frequentist hypothesis testing become biased towards rejecting the null hypothesis with sufficiently large samples?
I was just reading this article on the Bayes factor for a completely unrelated problem when I stumbled upon this passage
Hypothesis testing with Bayes factors is more robust than frequentist ...
- 691
54
votes
3
answers
2k
views
Do we have a problem of "pity upvotes"?
I know, this may sound like it is off-topic, but hear me out.
At Stack Overflow and here we get votes on posts, this is all stored in a tabular form.
E.g.:
post id voter id vote type ...
- 619
53
votes
3
answers
53k
views
Bootstrap vs. permutation hypothesis testing
There are several popular resampling techniques, which are often used in practice, such as bootstrapping, permutation test, jackknife, etc. There are numerous articles & books discuss these ...
- 2,977
53
votes
8
answers
75k
views
How can I test if given samples are taken from a Poisson distribution?
I know of normality tests, but how do I test for "Poisson-ness"?
I have sample of ~1000 non-negative integers, which I suspect are taken from a Poisson distribution, and I would like to test that.
- 1,321
53
votes
6
answers
17k
views
Motivation for Kolmogorov distance between distributions
There are many ways to measure how similar two probability distributions are. Among methods which are popular (in different circles) are:
the Kolmogorov distance: the sup-distance between the ...
- 3,206
52
votes
15
answers
7k
views
A smaller dataset is better: Is this statement false in statistics? How to refute it properly?
Dr. Raoult, who promotes Hydroxychloroquine, has some really intriguing statement about statistics in the biomedical field:
It's counterintuitive, but the smaller the sample size of a clinical test,...
- 674
52
votes
9
answers
13k
views
Statistical tests when sample size is 1
I'm a high school math teacher who is a bit stumped. A Biology student came to me with his experiment wanting to know what kind of statistical analysis he can do with his data (yes, he should have ...
- 631
52
votes
4
answers
2k
views
Cumming (2008) claims that distribution of p-values obtained in replications depends only on the original p-value. How can it be true?
I have been reading Geoff Cumming's 2008 paper Replication and $p$ Intervals: $p$ values predict the future only vaguely, but confidence intervals do much better [~200 citations in Google Scholar] -- ...
- 107k
51
votes
7
answers
7k
views
Why is "statistically significant" not enough?
I have completed my data analysis and got "statistically significant results" which is consistent with my hypothesis. However, a student in statistics told me this is a premature conclusion. Why? Is ...
- 621
48
votes
3
answers
72k
views
Test for bimodal distribution
I wonder if there is any statistical test to "test" the significance of a bimodal distribution. I mean, How much my data meets the bimodal distribution or not? If so, is there any test in the R ...
- 663
47
votes
5
answers
7k
views
Why is multiple comparison a problem?
I find it hard to understand what really is the issue with multiple comparisons. With a simple analogy, it is said that a person who will make many decisions will make many mistakes. So very ...
- 613
47
votes
4
answers
18k
views
Are smaller p-values more convincing?
I've been reading up on $p$-values, type 1 error rates, significance levels, power calculations, effect sizes and the Fisher vs Neyman-Pearson debate. This has left me feeling a bit overwhelmed. I ...
- 1,876
46
votes
4
answers
46k
views
Significance contradiction in linear regression: significant t-test for a coefficient vs non-significant overall F-statistic
I'm fitting a multiple linear regression model between 4 categorical variables (with 4 levels each) and a numerical output. My dataset has 43 observations.
Regression gives me the following $p$-...
- 2,634