# Tagged Questions

Statistical significance refers to the probability that, if, in the population from which this sample were drawn the true effect were 0 (or some hypothesized value) a test statistic as extreme or more extreme than the one gotten in the sample could have occurred.

1k views

### What are good references containing arguments against null hypothesis significance testing?

In the last few years I've read a number of papers arguing against the use of null hypothesis significance testing in science, but didn't think to keep a persistent list. A colleague recently asked me ...
2k views

### Regarding p-values, why 1% and 5%? Why not 6% or 10%?

Regarding p-values, I am wondering why 1% and 5% seem to be the gold standard for "statistical significance". Why not other values, like 6% or 10%? Is there a fundamental mathematical reason for ...
704 views

### What causes the discontinuity in the distribution of published p-values at p < .05?

In a recent paper , Masicampo and Lalande (M-L) collected a large number of p-values published in many different studies. They observed a curious jump in the histogram of the p-values right at the ...
3k views

### Do we need a global test before post hoc tests?

I often hear that post hoc tests after an ANOVA can only be used if the ANOVA itself was significant. However, post hoc tests adjust p values to keep the global type I error rate at 5%, don't they? So ...
4k views

### Correcting p values for multiple tests where tests are correlated (genetics)

I have p values from a lot of tests and would like to know whether there is actually something significant after correcting for multiple testing. The complication: my tests are not independent. The ...
62k views

### How to interpret and report eta squared / partial eta squared in statistically significant and non-significant analyses?

I have data that has eta squared values and partial eta squared values calculated as a measure of effect size for group mean differences. What is the difference between eta squared and partial eta ...
441 views

### Checking for a statistically significant peak

I have a set of data, $y$ and $x$. I would like to test the following hypothesis: There is a peak in $y$; that is as $x$ increases, $y$ first increases and then decreases. My first idea was fitting ...
17k views

### What is the difference between the Shapiro-Wilk test of normality and the Kolmogorov-Smirnov test of normality?

What is the difference between the Shapiro-Wilk test of normality and the Kolmogorov-Smirnov test of normality? When will results from these two methods differ?
1k views

### How do I compare bootstrapped regression slopes?

Let us assume I have two data sets with n observations of data pairs of independent variable x and dependent variable y each. Let us further assume I want to generate a distribution of regression ...
536 views

### Alternatives to p-values in R?

In a recent article discussing the demerits of relying on the p-value for statistical inference, called "Matrixx v. Siracusano and Student v. Fisher Statistical significance on trial" (DOI: ...
539 views

### Confidence interval and probability - where is the error in this statement?

If someone makes a statement like below: "Overall, nonsmokers exposed to environmental smoke had a relative risk of coronary heart disease of 1.25 (95 percent confidence interval, 1.17 to ...
3k views

### Assessing the significance of differences in distributions

I have two groups of data. Each with a different distribution of multiple variables. I'm trying to determine if these two groups' distributions are different in a statistically significant way. I ...
612 views

### What's the correct way to test the significance of classification results

There are many situations where you may train several different classifiers, or use several different feature extraction methods. In the literature authors often give the mean classification error ...
625 views

### Determining if change in a time series is statistically significant

I have the total number of calls received each week and have plotted them on a chart, going back nearly 3 years. By eye it seems that there was a massive drop over Christmas, that doesn't seem to ...
4k views

### How can one do an MCMC hypothesis test on a mixed effect regression model with random slopes?

The library languageR provides a method (pvals.fnc) to do MCMC significance testing of the fixed effects in a mixed effect regression model fit using lmer. However, pvals.fnc gives an error when the ...
517 views

### What does it mean for a study to be over-powered?

What does it mean for a study to be over-powered? My impression is that it means that your sample sizes are so large that you have the power to detect minuscule effect sizes. These effect sizes are ...
745 views

### Sequential hypothesis testing in basic science

I'm a pharmacologist and, in my experience, almost all papers in basic biomedical research use Student's t-test (either to support inference or to conform to expectations...). A couple of years ago it ...
336 views

### When/where to use functional data analysis?

I am very new to functional data analysis (FDA). I am reading: Ramsay, James O., and Silverman, Bernard W. (2006), Functional Data Analysis, 2nd ed., Springer, New York. However, I am still ...
4k views

### Comparing and contrasting, p-values, significance levels and type I error

I was wondering if anybody could give a concise rundown as to the definitions and uses of p-values, significance level and type I error. I understand that p-values are defined as "the probability of ...
3k views

### Either quadratic or interaction term is significant in isolation, but neither are together

As part of an assignment, I had to fit a model with two predictor variables. I then had to draw a plot of the models' residuals against one of the included predictors and make changes based on that. ...
4k views

### Examples of studies using p < 0.001, p < 0.0001 or even lower p-values?

I come from the social sciences, where p < 0.05 is pretty much the norm, with p < 0.1 and p < 0.01 also showing up, but I was wondering: what fields of study, if any, use lower p-values as a ...
618 views

### How do I find values not given in (/interpolate in) statistical tables?

Often people use programs to obtain p-values, but sometimes - for whatever reason - it may be necessary to obtain a critical value from a set of tables. Given a statistical table with a limited ...
344 views

### Test for proportions and binary classifier

I have a prototype machine producing parts. In a first test the machine produces $N_1$ parts and a binary classifier tells me that $d_1$ parts are defective ($d_1 < N_1$, usually $d_1/N_1<0.01$ ...
2k views

### What is the difference between “testing of hypothesis” and “test of significance”?

Is there a difference between the phrases "testing of hypothesis" and "test of significance" or are they the same? After a detailed answer from @Micheal Lew, I have one confusion that nowadays ...
674 views

### How to look for valleys in a graph?

I'm examining some genomic coverage data which is basically a long list (a few million values) of integers, each saying how well (or "deep") this position in the genome is covered. I would like to ...
395 views

### Is a p-value of 0.04993 enough to reject null hypothesis?

In a Wilcoxon signed-ranks statistical significance test, we came across some data that produces a $p$-value of $0.04993$. With a threshold of $p < 0.05$, is this result enough to reject the null ...
173 views

### Does significance test make sense to compare randomised groups at baseline?

Many randomised controlled trial (RCT) papers report significance tests on baseline parameters just after/before randomisation to show that the groups are indeed similar. This is often part of a ...
165 views

### Can the Mantel test be extended to asymmetric matrices?

The Mantel test is usually applied to symmetric distance/difference matrices. As far as I understand, an assumption of the test is that the measure used to define differences must be at least a ...
390 views

### Can one leave out data from research because it is not significant?

I've encountered this sentence while reading an article on sciencemag.org. In the end, responses from just 7600 researchers in 12 countries were included because the remaining data were not ...
2k views

### Do you reject the null hypothesis when $p < \alpha$ or $p \leq \alpha$?

This is clearly just a matter of definition or convention, and of almost no practical importance. If $\alpha$ is set to its traditional value of 0.05, is a $p$ value of 0.0500000000000... considered ...
3k views

### Cross-correlation significance in R

How do you tell if the correlations at different lags obtained from the cross-correlation (ccf function) of two time series are significant.
257 views

### Can you reproduce this chi-squared test result?

Over at Skeptics.StackExchange, an answer cites a study into electro-magnetic hypersensitivity: McCarty, Carrubba, Chesson, Frilot, Gonzalez-Toledo & Marino, Electromagnetic Hypersensitivity: ...
2k views

### What sense does it make to compare p-values to each other?

I have two populations, each containing 1000 samples. For each sample I have two properties A & B. I have used a t-test separately for A & B: both were found significantly different within the ...
2k views

### How can I test whether my clustering of binary data is significant

I'm doing shopping cart analyses my dataset is set of transaction vectors, with the items the products being bought. When applying k-means on the transactions, I will always get some result. A random ...
306 views

### Does an exact test always yield a higher P value then an approximated test?

I ran a simulation on that for mcnemar test, and the answer seemed to be yes. I was wondering if this can always be said to be the case that the exact P value is higher (or not smaller) then the p ...
748 views

### Statistically significant vs. independent/dependent

What is the difference between having something statistically significant (such as a difference between two samples) and stating if a group of numbers are independent or dependent.
1k views

### Finding the best features in interaction models

I have list of proteins with their feature values. A sample table looks like this: ...
320 views

### Why is running split tests until statistically significant a “bad thing”? (Or is it?)

I read this article on "how not to run an A/B test". And I still don't understand what exactly the author's reasoning is. Can someone dumb it down for me? I think what it might be saying is that ...
437 views

### How to quantify statistical insignificance?

I'm relatively new to statistics, and understand that my question may be completely misworded. I am testing my own algorithm versus another. While the outputs are not identical, I want to show that ...
782 views

### Safely determining sample size for A/B testing

I am a software engineer looking to build an A/B testing tool. I don't have a solid stats background but have been doing quite a bit of reading over the last few days. I am following the methodology ...
600 views

### What is a good index of the degree of violation of normality and what descriptive labels could be attached to that index?

Context: In a previous question, @Robbie asked in a study with around 600 cases why tests of normality suggested significant non-normality yet the plots suggested normal distributions. Several people ...
457 views

### Assessing significance of correlation

I have two variables, and I can calculate e.g. the Pearson correlation between them, but I would like to know something analogous to what a t-test would give me (i.e. some notion of how significant ...
226 views
+50

### How to test whether the variance of two distributions is different if the distributions are not normal

I'm studying two geographically-isolated populations of the same species. Inspecting the distributions, I see that both are bimodal (there's some seasonality to their occurrence), but the peaks in one ...
1k views

### Is the overlap between two gene expression samples significant?

I have performed an experiment to study the response of a yeast (that contains 5000 genes) to stress caused by heat shock. I have one list of 48 genes that are overexpressed at 37ºC and another list ...
439 views

### How to choose df for comparisons between summary statistics (e.g. slope values)?

In order to correlate or compare means of two dependent variables. In my case, I need to correlate individual (e.g. subjects=30) slope values from different conditions (e.g. conditions=4), and each ...
314 views

### Assessing statistical significance of a rare binary event in time series

I have a univariate time series which is an index of susceptibility to failure, and a binary variable which indicates whether a failure actually occurred in a given time window or not. I want to carry ...
118 views

### Can someone help me understand what type of problem I am looking at? Not sure if this classifies as hypothesis-testing

Please pardon me if this question is not clear. I am not sure if I am using the right terminologies. I have conducted an experiment in different environments multiple times. So my data looks ...
234 views

### Algorithm for determining performance speedup/slowdown in a code change vs. historical data?

I work on a large programming team, and I run a suite of performance tests on every change that is made in our program, which basically measure time it takes to run the test. For every code change, we ...
For discrete test statistics, the distribution of the corresponding $p$-value is discrete and stochastically larger than the uniform distribution. Hence the corresponding hypothesis test based on the ...