28

I don't think the objection is to just the term "statistically significant" but to the abuse of the whole concept of statistical significance testing and to the misinterpretation of results that are (or are not) statistically significant. In particular, look at these six statements: P-values can indicate how incompatible the data are with a specified ...


26

This is a great question; the answer depends a lot on context. In general I would say you are right: making an unqualified general claim like "group A used X more often than group B" is misleading. It would be better to say something like in our experiment group A used X more often than group B, but we're very uncertain how this will play out in the ...


24

I would argue that there is not any testing to do. If the sample correlation is not 1, then you reject $H_0: \rho=1$ with certainty. Having a correlation of 1 means that the points cannot deviate from a diagonal line the way that they can when $\vert \rho \vert < 1$. EDIT set.seed(2019) x <- rexp(1000) y <- 3*x plot(x,y) V <- rep(NA,10000) for ...


17

Assuming you have the original data and not just the summary of the fits, the general solution to this problem is to fit a model with an interaction, i.e. to go back to the data and fit the model $$ Y = \beta_0 + \beta_1 I(t>t_I) + \beta_2 (t-t_I) + \beta_3 I(t>t_I) (t-t_I) $$ where $I(t>t_I)$ is an indicator variable, i.e. =1 if $t>t_I$ and 0 ...


15

Here is my paraphrase of what Fisher says in your bolded quote. It should not be forgotten that quite a lot goes into choosing what hypothesis to test, so much so that even for a single person's decision, you could not specify it all. It also should not be forgotten that, for reasons stated above, you cannot decide on a particular trial's significance level ...


14

For the purpose of this answer I'm going to assume that excluding those few participants was fully justified, but I agree with Patrick that this is a concern. There's no meaningful difference between p ~ 0.05 or p = 0.06. The only difference here is that the convention is to treat the former as equivalent to 'true' and the latter as equivalent to 'false'. ...


11

A few general points before answering the individual questions. First, in logistic regression (unlike in linear regression) coefficient estimates will be biased if you omit any predictor associated with outcome whether or not it is correlated with the included predictors. This page gives an analytic demonstration for the related probit regression. Second, ...


10

I can hardly imagine any worthwhile effect size that requires such a large sample size to be decently powered. There's no "bias" of having unequal sample sizes$^1$. The only disadvantage is that the power of the test tends to be somewhat limited by the smaller group. For even very small effects, 30,000 observations may confer quite a powerful test. $^1$ ...


10

The cases to which Fisher is referring are not observations but tests. That is, we select hypotheses to test. We don't just test random hypotheses - we base them on observation, the literature, scientific theories and so on. If you did test random hypotheses, then the number of times you are mistaken (in the first sentence of your quote) would be 1% (or ...


10

A couple of points: Even there are exceptions, the general rule involving interactions is that you need to include the lower order terms. That is, in a model in which you want to include a 3-way interaction, you also need to include the main effects, and all 2-way interactions. In that regard, model mod2 does that. In general interactions are complex terms ...


10

One straightforward way to analyze this situation is to assume for testing purposes that the fix made no difference. Under this assumption, you may view the assignment of the (potential) 48 observations into pre-fix and post-fix groups as being random. If (hypothetically) all the post-fix outcomes work as expected, it means you have observed 44 expected ...


10

The F-test specifically examines for differences in variance and does not need to be sensitive to other differences such as mean. KS has to be sensitive to every kind of distributional difference, whether that difference is mean, variance, or multimodality. Think of the F-test as a specialist that will be great at finding differences in variance at the ...


9

If you want to quantify departure from normality, then a good measure is the Kolmogorov-Smirnov test statistic $D.$ Let's compare two samples of size $n = 5000.$ The sample x below it taken using an excellent algorithm in R that is known to sample from an essentially perfect normal population, $\mathsf{Norm}(\mu=1.5, \sigma=0.5).$ The sample y is based on ...


8

Maybe a couple of examples will help to illustrate some of the issues. Suppose the two populations are $X \sim \mathsf{Norm}(\mu = 500, \sigma =30)$ and $Y \sim \mathsf{Norm}(\mu = 501, \sigma = 20.)$ If both sample sizes are $150,000,$ then there is sufficient power to detect the small difference in means. set.seed(422) x = rnorm(150000, 500, 30) y = ...


8

There are an awful lot of issues raised in your question, so I will try to give answers on each of the issues you raise. To frame some of these issues clearly, it is important to note at the outset that a p-value is a continuous measure of evidence against the null hypothesis (in favour of the stated alternative), but when we compare it to a stipulated ...


8

This is the problem of rare event estimation. The number of shots made by player $i$ is a Binomial random variable $$X_i \sim Binom(1000, p_i)$$ where $p_i \approx 1/1000$ is very small. In general, $1000$ balls is not nearly enough to accurately estimate $p_i$. The problem is that the likelihood function has too much "overlap" for different values of $x_i$....


8

In my opinion one of more honest yet non-technical phrasing would be something like: The obtained result is surprising/unexpected (p = 0.03) under the assumption of no mean difference between the groups. Or, permitting the format, it could be expanded: The obtained difference of $\Delta m$ would be quite surprising (p = 0.03) under the scenario of two ...


7

I don't think it is the same. If you say that $\hat \beta$ is statistically significant, that's a short way to say that it's significantly different from 0, and "different from 0" is not the same as "greater than 0", obviously. If I read: $\hat \beta$ is statistically significantly positive I understand that it has been tested for being greater than 0 (...


7

Significance testing consists of defining a rejection region, and rejecting if the data is in that region. The size of the region is its $\alpha$ value. If two different regions are different shapes, then even if one is smaller than the other, there can be places that are inside the smaller one but not in the larger one. Dave’s answer explains that KS tests ...


7

The essential features of this question are: It does not make strong distributional assumptions, lending it a non-parametric flavor. It concerns only tail behavior, not the entire distribution. With some diffidence--because I have not studied my proposal theoretically to fully understand its performance--I will outline an approach that might be practicable....


6

The chi-squared test requires the counts because those determine how uncertain each of the proportions is. However, it's still possible to make some progress. For instance, if the value of 19% represents 190 out of 1000 observed in a random sample, its standard error is only 1.24%; but if it represents 3 out of 16 observed, its standard error is 9.8%. ...


6

The underlying difficulty is that cross validation results (actually: all test results) are subject to several sources of variance (read the Dietterich and Yoshua & Bengio papers). The usual tests the linked blog post discusses all assume that the data can be described using one variance term. Sources of variance: We're calculating the test results ...


6

That x is probably representing an interaction term. You should really give us more information about the data (and the experiment which produced it), but the SS's here seems to be sequential SS's, as $0.668+0.357+1.078 =2.103$. Note also that for the df's (degrees of freedom) from the table we have: $119-78-2-39=0$, so there is no df's left for error. ...


6

Yes, it's completely possible to get large coefficients and small $R^2$ values. Consider that the coefficient values depend on units, while $R^2$ does not. So if a model with length as a predictor switches from using kilometres to millimetres, the coefficients will increase by a factor of $10^6$ without changing the $R^2$. Yes, your interpretation is ...


6

I think it's been asked before. It's useful to realize that, without a prespecified sample size and alpha level, the $p$-value is just a measure of the sample size you ultimately wind up with. Not appealing. An approach I use is this: at what sample size would a 0.05 level be appropriate? Scale accordingly. For instance, I feel the 0.05 level is often suited ...


6

You are right that the documentation is wrong. Note that p values are defined somewhat differently from what you write. They do not measure the probability of a decision, such as the decision to reject the null or to fail to reject the null. They measure the probability of test statistics. Whether or not to reject a null hypothesis is a subsequent decision ...


6

No, you can't. The Kolmogorov-Smirnov test is used to test whether two distributions are equal. What you are interested in is whether two distributions have the same mean. For instance, two distributions $(1,2,3)$ and $(2,2,2)$ are unequal, and the KS test would detect this (except for the sample size, which is too small), but their means are equal, so the ...


6

It is actually used quite often (in fact, Bonferroni and Bonferroni-Holm can be shown to be valid tests using the closed testing principle - see below). Part of the reason why some simple procedures like Bonferroni are still so popular is of course, that they are very easy to implement and it is easy to communicate what you did. E.g. even for the simple ...


6

To start off, you shouldn't be using backwards selection at all if you believe an interaction effect to be present. If the full model is the model you presumed, then its coefficients are the only interesting ones. Also note that the results of these models do not conflict with each other: The marginal effects do not have the same interpretation as the main ...


6

I agree with the answer by Peter Flom, but would like to add an additional point on the use of the term "significance" in statistical hypothesis testing. Most hypothesis tests of interest in statistics have a null hypothesis that posits a zero value for some "effect" and an alternative hypothesis that posits a non-zero (or positive, or negative) value for ...


Only top voted, non community-wiki answers of a minimum length are eligible