Statistical significance refers to the probability that, if, in the population from which this sample were drawn the true effect were 0 (or some hypothesized value) a test statistic as extreme or more extreme than the one gotten in the sample could have occurred.
33
votes
2answers
1k views
What are good references containing arguments against null hypothesis significance testing?
In the last few years I've read a number of papers arguing against the use of null hypothesis significance testing in science, but didn't think to keep a persistent list. A colleague recently asked me ...
24
votes
5answers
955 views
Regarding p-values, why 1% and 5%? Why not 6% or 10%?
Regarding p-values, I am wondering why 1% and 5% seem to be the gold standard for "statistical significance". Why not other values, like 6% or 10%?
Is there a fundamental mathematical reason for ...
18
votes
3answers
560 views
What causes the discontinuity in the distribution of published p-values at p < .05?
In a recent paper , Masicampo and Lalande (M-L) collected a large number of p-values published in many different studies. They observed a curious jump in the histogram of the p-values right at the ...
15
votes
2answers
2k views
Do we need a global test before post hoc tests?
I often hear that post hoc tests after an ANOVA can only be used if the ANOVA itself was significant. However, post hoc tests adjust p values to keep the global type I error rate at 5%, don't they? So ...
14
votes
4answers
3k views
Correcting p values for multiple tests where tests are correlated (genetics)
I have p values from a lot of tests and would like to know whether there is actually something significant after correcting for multiple testing. The complication: my tests are not independent. The ...
14
votes
2answers
385 views
Checking for a statistically significant peak
I have a set of data, $y$ and $x$. I would like to test the following hypothesis: There is a peak in $y$; that is as $x$ increases, $y$ first increases and then decreases.
My first idea was fitting ...
12
votes
1answer
41k views
How to interpret and report eta squared / partial eta squared in statistically significant and non-significant analyses?
I have data that has eta squared values and partial eta squared values calculated as a measure of effect size for group mean differences.
What is the difference between eta squared and partial eta ...
12
votes
4answers
477 views
Alternatives to p-values in R?
In a recent article discussing the demerits of relying on the p-value for statistical inference, called "Matrixx v. Siracusano and Student v. Fisher
Statistical significance on trial" (DOI: ...
11
votes
3answers
893 views
How do I compare bootstrapped regression slopes?
Let us assume I have two data sets with n observations of data pairs of independent variable x and dependent variable y each. Let us further assume I want to generate a distribution of regression ...
11
votes
1answer
497 views
Determining if change in a time series is statistically significant
I have the total number of calls received each week and have plotted them on a chart, going back nearly 3 years.
By eye it seems that there was a massive drop over Christmas, that doesn't seem to ...
11
votes
1answer
12k views
What is the difference between the Shapiro-Wilk test of normality and the Kolmogorov-Smirnov test of normality?
What is the difference between the Shapiro-Wilk test of normality and the Kolmogorov-Smirnov test of normality? When will results from these two methods differ?
11
votes
1answer
298 views
When/where to use functional data analysis?
I am very new to functional data analysis (FDA). I am reading:
Ramsay, James O., and Silverman, Bernard W. (2006), Functional Data
Analysis, 2nd ed., Springer, New York.
However, I am still ...
10
votes
5answers
465 views
Confidence interval and probability - where is the error in this statement?
If someone makes a statement like below:
"Overall, nonsmokers exposed to environmental smoke had a relative
risk of coronary heart disease of 1.25 (95 percent confidence
interval, 1.17 to ...
10
votes
5answers
2k views
Assessing the significance of differences in distributions
I have two groups of data. Each with a different distribution of multiple variables. I'm trying to determine if these two groups' distributions are different in a statistically significant way. I ...
10
votes
5answers
358 views
What's the correct way to test the significance of classification results
There are many situations where you may train several different classifiers, or use several different feature extraction methods. In the literature authors often give the mean classification error ...
10
votes
1answer
305 views
Test for proportions and binary classifier
I have a prototype machine producing parts.
In a first test the machine produces $N_1$ parts and a binary classifier tells me that $d_1$ parts are defective ($d_1 < N_1$, usually $d_1/N_1<0.01$ ...
10
votes
1answer
675 views
Sequential hypothesis testing in basic science
I'm a pharmacologist and, in my experience, almost all papers in basic biomedical research use Student's t-test (either to support inference or to conform to expectations...). A couple of years ago it ...
9
votes
4answers
3k views
Examples of studies using p < 0.001, p < 0.0001 or even lower p-values?
I come from the social sciences, where p < 0.05 is pretty much the norm, with p < 0.1 and p < 0.01 also showing up, but I was wondering: what fields of study, if any, use lower p-values as a ...
9
votes
4answers
569 views
How to look for valleys in a graph?
I'm examining some genomic coverage data which is basically a long list (a few million values) of integers, each saying how well (or "deep") this position in the genome is covered.
I would like to ...
9
votes
2answers
3k views
How can one do an MCMC hypothesis test on a mixed effect regression model with random slopes?
The library languageR provides a method (pvals.fnc) to do MCMC significance testing of the fixed effects in a mixed effect regression model fit using lmer. However, pvals.fnc gives an error when the ...
9
votes
3answers
442 views
What does it mean for a study to be over-powered?
What does it mean for a study to be over-powered?
My impression is that it means that your sample sizes are so large that you have the power to detect minuscule effect sizes. These effect sizes are ...
9
votes
1answer
153 views
Does significance test make sense to compare randomised groups at baseline?
Many randomised controlled trial (RCT) papers report significance tests on baseline parameters just after/before randomisation to show that the groups are indeed similar. This is often part of a ...
9
votes
2answers
241 views
Can you reproduce this chi-squared test result?
Over at Skeptics.StackExchange, an answer cites a study into electro-magnetic hypersensitivity:
McCarty, Carrubba, Chesson, Frilot, Gonzalez-Toledo & Marino, Electromagnetic Hypersensitivity: ...
8
votes
4answers
377 views
Can one leave out data from research because it is not significant?
I've encountered this sentence while reading an article on sciencemag.org.
In the end, responses from just 7600 researchers in 12 countries were included because the remaining data were not ...
8
votes
3answers
2k views
Comparing and contrasting, p-values, significance levels and type I error
I was wondering if anybody could give a concise rundown as to the definitions and uses of p-values, significance level and type I error.
I understand that p-values are defined as "the probability of ...
8
votes
2answers
1k views
Do you reject the null hypothesis when $p < \alpha$ or $p \leq \alpha$?
This is clearly just a matter of definition or convention, and of almost no practical importance. If $\alpha$ is set to its traditional value of 0.05, is a $p$ value of 0.0500000000000... considered ...
8
votes
0answers
121 views
Can the Mantel test be extended to asymmetric matrices?
The Mantel test is usually applied to symmetric distance/difference matrices. As far as I understand, an assumption of the test is that the measure used to define differences must be at least a ...
8
votes
2answers
1k views
What sense does it make to compare p-values to each other?
I have two populations, each containing 1000 samples. For each sample I have two properties A & B. I have used a t-test separately for A & B: both were found significantly different within the ...
7
votes
3answers
1k views
How can I test whether my clustering of binary data is significant
I'm doing shopping cart analyses my dataset is set of transaction vectors, with the items the products being bought.
When applying k-means on the transactions, I will always get some result. A random ...
7
votes
2answers
261 views
Does an exact test always yield a higher P value then an approximated test?
I ran a simulation on that for mcnemar test, and the answer seemed to be yes.
I was wondering if this can always be said to be the case that the exact P value is higher (or not smaller) then the p ...
7
votes
2answers
588 views
Statistically significant vs. independent/dependent
What is the difference between having something statistically significant (such as a difference between two samples) and stating if a group of numbers are independent or dependent.
7
votes
1answer
288 views
Why is running split tests until statistically significant a “bad thing”? (Or is it?)
I read this article on "how not to run an A/B test".
And I still don't understand what exactly the author's reasoning is. Can someone dumb it down for me?
I think what it might be saying is that ...
7
votes
1answer
360 views
How to quantify statistical insignificance?
I'm relatively new to statistics, and understand that my question may be completely misworded. I am testing my own algorithm versus another. While the outputs are not identical, I want to show that ...
7
votes
2answers
482 views
Safely determining sample size for A/B testing
I am a software engineer looking to build an A/B testing tool. I don't have a solid stats background but have been doing quite a bit of reading over the last few days.
I am following the methodology ...
7
votes
2answers
1k views
What is the difference between “testing of hypothesis” and “test of significance”?
Is there a difference between the phrases "testing of hypothesis" and "test of significance" or are they the same?
After a detailed answer from @Micheal Lew, I have one confusion that nowadays ...
7
votes
2answers
418 views
Assessing significance of correlation
I have two variables, and I can calculate e.g. the Pearson correlation between them, but I would like to know something analogous to what a t-test would give me (i.e. some notion of how significant ...
7
votes
3answers
422 views
How to choose df for comparisons between summary statistics (e.g. slope values)?
In order to correlate or compare means of two dependent variables.
In my case, I need to correlate individual (e.g. subjects=30) slope values from different conditions (e.g. conditions=4), and each ...
7
votes
2answers
728 views
Is the overlap between two gene expression samples significant?
I have performed an experiment to study the response of a yeast (that contains 5000 genes) to stress caused by heat shock. I have one list of 48 genes that are overexpressed at 37ºC and another list ...
7
votes
1answer
282 views
Assessing statistical significance of a rare binary event in time series
I have a univariate time series which is an index of susceptibility to failure, and a binary variable which indicates whether a failure actually occurred in a given time window or not. I want to carry ...
7
votes
1answer
115 views
Can someone help me understand what type of problem I am looking at? Not sure if this classifies as hypothesis-testing
Please pardon me if this question is not clear. I am not sure if I am using the right terminologies.
I have conducted an experiment in different environments multiple times. So my data looks ...
7
votes
2answers
828 views
Hypothesis testing and significance for time series
A usual test of significance when looking a two populations is the t-test, paired t-test if possible. This assumes that the distribution is normal.
Are there similar simplifying assumptions that ...
7
votes
1answer
212 views
Statistics for gambling machine validation
Problem is that government wants to close electronic roulette and they claim that roulette failed at statistical test.
Sorry for my language but this is translated from Slovenian law as good as ...
7
votes
1answer
112 views
Conservativeness of Tests Based on Discrete Random Variables
For discrete test statistics, the distribution of the corresponding $p$-value is discrete and stochastically larger than the uniform distribution. Hence the corresponding hypothesis test based on the ...
6
votes
3answers
311 views
Why does this excerpt say that unbiased estimation of standard deviation usually isn't relevant?
I was reading on the computation of the unbiased estimation of standard deviation and the source I read stated
(...) except in some important situations, the task has little relevance to
...
6
votes
3answers
4k views
How to test group differences on a five point variable?
I have a series of observations that fall into bins (or "scores"); that is, the data can be 0, 1, 2, 3 or 4. There are two groups of such data, control and treated. I know the number of individuals ...
6
votes
3answers
204 views
A Stats 101 question with a real world application
This is probably a far too basic question for this board - but on the other hand, I know I'll get good answers. "Stats 101" is a metaphor, by the way. I'm asking for help with my work, not my ...
6
votes
1answer
465 views
What is a good index of the degree of violation of normality and what descriptive labels could be attached to that index?
Context:
In a previous question, @Robbie asked in a study with around 600 cases why tests of normality suggested significant non-normality yet the plots suggested normal distributions. Several people ...
6
votes
2answers
1k views
Statistical significance of changes over time on a 5-point Likert item
Context:
I have two data sets from the same questionnaire run over two years. Each question is measured using a 5-Likert scale.
Q1: Coding scheme
At the moment, I have coded my responses on a [0, ...
6
votes
2answers
1k views
Poisson distribution and statistical significance
Lets say I have a website which gets 100 hits per day (mu = 100). Yesterday my website got 130 hits (x = 130). If I assume a Poisson distribution, then the probability of getting 130 hits is:
...
6
votes
2answers
631 views
Finding the best features in interaction models
I have list of proteins with their feature values. A sample table looks like this:
...
