79

If you check the references below you'll find quite a bit of variation in the background, though there are some common elements. Those numbers are at least partly based on some comments from Fisher, where he said (while discussing a level of 1/20) It is convenient to take this point as a limit in judging whether a deviation is to be considered ...


76

EXECUTIVE SUMMARY: if "p-hacking" is to be understood broadly a la Gelman's forking paths, the answer to how prevalent it is, is that it is almost universal. Andrew Gelman likes to write about this topic and has been posting extensively about it lately on his blog. I don't always agree with him but I like his perspective on $p$-hacking. Here is an excerpt ...


70

This is certainly a valid concern, but this isn't quite right. If 1,000,000 studies are done and all the null hypotheses are true then approximately 50,000 will have significant results at p < 0.05. That's what a p value means. However, the null is essentially never strictly true. But even if we loosen it to "almost true" or "about right" or some such,...


69

Humor is a very personal thing - some people will find it amusing, but it may not be funny to everyone - and attempts to explain what makes something funny often fail to convey the funny, even if they explain the underlying point. Indeed not all xkcd's are even intended to be actually funny. Many do, however make important points in a way that's thought ...


65

The first three points, as far as I can tell, are a variation on a single argument. Scientists often treat uncertainty measurements ($12 \pm 1 $, for instance) as probability distributions that look like this: When actually, they are much more likely to look like this: As a former chemist, I can confirm that many scientists with non-mathematical ...


62

In addition to @gung's answer, I'll try to provide an example of what the anova function actually tests. I hope this enables you to decide what tests are appropriate for the hypotheses you are interested in testing. Let's assume that you have an outcome $y$ and 3 predictor variables: $x_{1}$, $x_{2}$, and $x_{3}$. Now, if your logistic regression model ...


54

I would like to list another intuitive example. Suppose I tell you I can predict the outcome of any coin flip. You do not believe and want to test my ability. You tested 5 times, and I got all of them right. Do you believe I have the special ability? Maybe not. Because I can get all of them right by chance. (Specifically, suppose the coin is a fair coin, ...


53

Hypothesis testing versus parameter estimation Typically, hypotheses are framed in a binary way. I'll put directional hypotheses to one side, as they don't change the issue much. It is common, at least in psychology, to talk about hypotheses such as: the difference between group means is or is not zero; the correlation is or is not zero; the regression ...


46

In the R package AER you will find the function dispersiontest, which implements a Test for Overdispersion by Cameron & Trivedi (1990). It follows a simple idea: In a Poisson model, the mean is $E(Y)=\mu$ and the variance is $Var(Y)=\mu$ as well. They are equal. The test simply tests this assumption as a null hypothesis against an alternative where $...


45

Use pt and make it two-tailed. > 2*pt(11.244, 30, lower=FALSE) [1] 2.785806e-12


45

Traditionally, the null hypothesis is a point value. (It is typically $0$, but can in fact be any point value.) The alternative hypothesis is that the true value is any value other than the null value. Because a continuous variable (such as a mean difference) can take on a value which is indefinitely close to the null value but still not quite equal and ...


45

Think of the data as the tip of the iceberg - all you can see above the water is the tip of the iceberg but in reality you are interested in learning something about the entire iceberg. Statisticians, data scientists and others working with data are careful to not let what they see above the water line influence and bias their assessment of what's hidden ...


40

Aren't all researches around the world somewhat like the "infinite monkey theorem" monkeys? Remember, scientists are critically NOT like infinite monkeys, because their research behavior--particularly experimentation--is anything but random. Experiments are (at least supposed to be) incredibly carefully controlled manipulations and measurements that are ...


38

The solution is a simple google away: http://en.wikipedia.org/wiki/Statistical_hypothesis_testing So you would like to test the following null hypothesis against the given alternative $H_0:p_1=p_2$ versus $H_A:p_1\neq p_2$ So you just need to calculate the test statistic which is $$z=\frac{\hat p_1-\hat p_2}{\sqrt{\hat p(1-\hat p)\left(\frac{1}{n_1}+\...


37

I have to give a non-answer (same as here): "... surely, God loves the .06 nearly as much as the .05. Can there be any doubt that God views the strength of evidence for or against the null as a fairly continuous function of the magnitude of p?" (p.1277) Rosnow, R. L., & Rosenthal, R. (1989). Statistical procedures and the justification of ...


35

You've interpreted the test wrong. If the p value is greater than 0.05 then the residuals are independent which we want for the model to be correct. If you simulate a white noise time series using the code below and use the same test for it then the p value will be greater than 0.05. m = c(ar, ma) w = arima.sim(m, 120) w = ts(w) plot(w) Box.test(w, type="...


32

You have gotten several good answers already. There are reasons to keep covariates and reasons to drop covariates. Statistical significance should not be a key factor, in the vast majority of cases. Covariates may be of such substantive importance that they have to be there. The effect size of a covariate may be high, even if it is not significant. The ...


31

In theory, yes... The results of individual studies may be insignificant but viewed together, the results may be significant. In theory you can proceed by treating the results $y_i$ of study $i$ like any other random variable. Let $y_i$ be some random variable (eg. the estimate from study $i$). Then if $y_i$ are independent and $E[y_i]=\mu$, you can ...


30

Yes. Suppose you have $N$ p-values from $N$ independent studies. Fisher's test (EDIT - in response to @mdewey's useful comment below, it is relevant to distinguish between different meta tests. I spell out the case of another meta test mentioned by mdewey below) The classical Fisher meta test (see Fisher (1932), "Statistical Methods for Research Workers" ...


30

Yes! That a coefficient is statistically indistinguishable from zero does not imply that the coefficient actually is zero, that the coefficient is irrelevant. That an effect does not pass some arbitrary cutoff for statistical significance does not imply one should not attempt to control for it. Generally speaking, the problem at hand and your research ...


29

Consider the case where the null hypothesis is that a coin is 2 headed, i.e. the probability of heads is 1. Now the data is the result of flipping a coin a single time and seeing heads. This results in a p-value of 1.0 which is greater than every reasonable alpha. Does this mean that the coin is 2 headed? it could be, but it could also be a fair coin and ...


29

Clearly I don't need to tell you what a p-value is, or why over-reliance on them is a problem; you apparently understand those things quite well enough already. With publishing, you have two competing pressures. The first - and one you should push for at every reasonable opportunity - is to do what makes sense. The second, ultimately, is the need to ...


29

Think about it in terms of proportions. Let's say that preferring an orange is a success, while preferring an apple is a failure. So your mean success rate is $\mu = \frac{\text{# of sucesses}}{n}$ or in this case .6 The standard error of this quantity is estimated to be $\sqrt{\frac{\mu(1-\mu)}{n}}$. For a small sample size (i.e. 10), the standard error is ...


28

You are correct that those $p$-values only tell you whether each level's mean is significantly different from the reference level's mean. Therefore, they only tell you about the pairwise differences between the levels. To test whether the categorical predictor, as a whole, is significant is equivalent to testing whether there is any heterogeneity in the ...


28

I don't think the objection is to just the term "statistically significant" but to the abuse of the whole concept of statistical significance testing and to the misinterpretation of results that are (or are not) statistically significant. In particular, look at these six statements: P-values can indicate how incompatible the data are with a specified ...


27

This answer is in two main parts: firstly, using linear interpolation, and secondly, using transformations for more accurate interpolation. The approaches discussed here are suitable for hand calculation when you have limited tables available, but if you're implementing a computer routine to produce p-values, there are much better approaches (if tedious when ...


26

The long answer is "yes". There are few reasons to remove insignificant predictors and many reasons not to. As far as interpreting them you do so ignoring the $P$-value just as you might interpret other predictors: with confidence intervals for effects over interesting ranges of the predictor.


26

Just to add to the existing answers (which are great, by the way). It is important to be aware that statistical significance is a function of sample size. When you get more and more data, you can find statistically significant differences wherever you look. When the amount of data is huge, even the tiniest effects can lead to statistical significance. This ...


26

If you are looking for an 'exact' test for two binomial proportions, I believe you are looking for Fisher's Exact Test. In R it is applied like so: > fisher.test(matrix(c(17, 25-17, 8, 20-8), ncol=2)) Fisher's Exact Test for Count Data data: matrix(c(17, 25 - 17, 8, 20 - 8), ncol = 2) p-value = 0.07671 alternative hypothesis: true odds ratio is not ...


26

We use these tests for different reasons and under different circumstances. $z$-test. A $z$-test assumes that our observations are independently drawn from a Normal distribution with unknown mean and known variance. A $z$-test is used primarily when we have quantitative data. (i.e. weights of rodents, ages of individuals, systolic blood pressure, etc.) ...


Only top voted, non community-wiki answers of a minimum length are eligible