34 votes
Accepted

Interpretation of p-value near alpha level

There are two different approaches to interpreting statistical significance - the Fisher way, and the Neyman-Pearson way. We smush these together (into what Gerd Gigerenzer has called a 'bastardised ...
user avatar
  • 14.8k
22 votes
Accepted

Can an irrelevant variable be significant in a regression model?

Here is a good example. Suppose you are interested in modelling the effect of ice cream sales on incidence of shark attacks. Now, clearly there is no association; buying ice cream in no way affects ...
user avatar
19 votes

Bayesian criticism of frequentist p-value

The point that the authors are trying to make is a subtle one: they see it as a failure of NHST that, as $n$ gets arbitrarily large, the $p$-value doesn't tend to 1. It's a bit surprising that this ...
user avatar
  • 53.2k
19 votes
Accepted

Frequentist perspective of regression coefficients and significance (coming from Bayesian background)?

A p value is the probability of observing a test statistic as or more extreme than the researcher's own test statistic, assuming the null hypothesis, and an assumed distribution model are both true. ...
user avatar
  • 26.6k
12 votes

(Sudden deafness ended?) How can I better communicate -and did I find- a major (conclusion-flipping) statistical error in a recently published paper?

It's exciting to think that one has seen a fatal flaw in some published work. But sometimes that vision of a "fatal flaw" is a mirage, arising from a misunderstanding of the question at hand,...
user avatar
  • 63.8k
12 votes

Hypothesis testing: why $\mu > 0$ (or even $\mu > \epsilon$) "seems easier” to substantiate than $\mu \neq 0$?

There are some misconceptions in your question that I need to clear up before getting to the answer. The null hypothesis $H_0$ in a statistical test is always the claim you want to argue against. The ...
user avatar
  • 31.4k
11 votes
Accepted

What can cause exploding statistic values and p-values near zero with the Wilcoxon-Mann-Whitney test?

Nothing out of the ordinary is going on from the sound of it. In almost all cases, I get huge values for the statistic Have you looked at the range of possible values for the statistic? For the ...
user avatar
  • 261k
11 votes
Accepted

Comparing model evaluations of machine learning and statistics

Is it true to some extent that statisticians are usually more concerned about the model's goodness-of-fit and the corresponding metrics of significance, and not that much about model's generalization ...
user avatar
  • 587
9 votes

(Sudden deafness ended?) How can I better communicate -and did I find- a major (conclusion-flipping) statistical error in a recently published paper?

I write in respect to *. This study should be withdrawn. I'm not going to wade through the material here to form a view on whether you are right or wrong in your identification of an error. However, ...
user avatar
  • 96.6k
9 votes

Correct use of Chi-square and hypothesis testing?

All of your concerns are valid and well articulated. The chi-square test simply provides the weight of the evidence that there is an association between attendance and purchase. Any causality is an ...
user avatar
8 votes

Bayesian criticism of frequentist p-value

I think you’re conflating two different arguments against p-values. Let me spell them out. By definition, p is distributed uniformly under the null hypothesis (or as uniformly as possible in discrete ...
user avatar
  • 1,254
8 votes
Accepted

Is it still necessary to correct for multiple comparisons/testing if the sample sizes are large?

Think buying hundreds of fair dice. You do not know they are, though, and hence test if each has an expected value of 3.5 points, via throwing each many times (1000+). One of them must come up as &...
user avatar
8 votes

What is the history of $p < 0.05$ or 95% confidence?

Fisher suggested the 0.05 level indirectly. He mentioned that two standard deviations is an easy rule for significance, and the 0.05 level is what approximately corresponds to it. From Fisher's 1925 '...
user avatar
8 votes

Statistically compare two large continuous datasets

One feature (not a bug) of hypothesis testing is that it gets more sensitive to small differences as the sample size increases. Consequently, hypothesis testing considers more than just effect size, ...
user avatar
  • 31.7k
8 votes

Can an irrelevant variable be significant in a regression model?

"Relevant" and "irrelevant" are not well-defined statistical terms that everyone agrees on. Case in point: the answer by @Demetri Pananos interprets "relevant" as "...
user avatar
  • 2,511
8 votes

Can an irrelevant variable be significant in a regression model?

I think a variable can be irrelevant and significant at the same time. But, how do I explain that? This can be explained by using the concept of type I errors. Below is an example by repeating a t-...
user avatar
7 votes

Likelihood of Global Catastrophe - Surely it cant be 80%?

Check the reference for the figure mentioned by the report: Sandberg, A. & Bostrom, N. (2008): “Global Catastrophic Risks Survey”, Technical. Report #2008-1, Future of Humanity Institute, Oxford ...
user avatar
  • 114k
7 votes
Accepted

What is the concepts of nominal and actual significance level?

Note: I suspect that there are at least two different meanings of "actual significance level" around, but here's one that makes sense to me: The nominal significance level is the ...
user avatar
7 votes
Accepted

What is the parameter in Spearman's $\rho$

Suppose $(X_1,Y_1),(X_2,Y_2),\ldots,(X_n,Y_n)$ are i.i.d random vectors with a continuous distribution. Let $R_i =\operatorname{Rank}(X_i)$ among $X_1,X_2,\ldots,X_n$ and $Q_i=\operatorname{Rank}(Y_i)$...
user avatar
  • 9,170
7 votes

Comparing model evaluations of machine learning and statistics

The answer is no, too many claims in your post, not only to the main question. The difference between ML and stats is arbitrary, superficial, and not important. There is plenty of statisticians that ...
user avatar
  • 6,156
7 votes

What is the exact mathematical relationship between the p-values and t-stat/degrees of freedom?

Accurate p-values for t-tests are not really feasible "by hand"; note that you can work out p-values from the cdf, so I will focus on that. At very low, integer degrees of freedom, the cdf ...
user avatar
  • 261k
7 votes
Accepted

Can dropping an insignificant factor from a model make the model worse?

In this case you are relying on the wrong test to decide that Zone is not significant. Note that the coefficients of the Zone effect are large (>30) with huge standard errors. This happens when the ...
user avatar
  • 10.3k
7 votes

How to report a P-value?

When is it appropriate to give the exact P-value, instead of writing e.g. P<0.05 (also in case of non-significant P-values)? As a general guideline you want to convey as much information about ...
user avatar
7 votes
Accepted

Linear relationship seems to depends from one value: would bootstraping help?

First off: If I extend this argument, would removing other value also impact the relationship? Almost certainly yes. The regression line slope depends on all observations, so removing any point will ...
user avatar
6 votes

Regarding p-values, why 1% and 5%? Why not 6% or 10%?

A recent article sheds some light on the arbitrariness of $p$-values; the selection of two thresholds was motivated, at least in part, as a work-around to a dispute over publishing rights. Briefly, ...
user avatar
  • 79.4k
6 votes

Bayesian criticism of frequentist p-value

For me a core issue here is that the Bayesian criticism of the p-value is based on Bayesian reasoning that a frequentist would not normally accept. For the Bayesian, the "true parameter" is ...
user avatar
6 votes
Accepted

T-tests provide info about the margin of mean difference?

if I can use the t-test to claim that one dataset provides significantly better results than the other I wanted to point out that the expression "statistically significant" has been ...
user avatar
  • 3,110
6 votes

Hypothesis testing: why $\mu > 0$ (or even $\mu > \epsilon$) "seems easier” to substantiate than $\mu \neq 0$?

Hypothesis testing: why $\mu > 0$ (or even $\mu > \epsilon$) "seems easier” to substantiate than $\mu \neq 0$? It seems easier because the one-sided t-test and two-sided t-test have ...
user avatar
6 votes

Hypothesis testing: why $\mu > 0$ (or even $\mu > \epsilon$) "seems easier” to substantiate than $\mu \neq 0$?

the data may strongly support $\mu > 0$, but does not constitute sufficient evidence for $\mu \neq 0$ I don't know if this reasoning helps or if it is 100% correct... You accept to be wrong $\...
user avatar
  • 3,110
6 votes

If I have one non-significant factor level in a glm, is that entire variable now considered non significant?

When you have a factor variable in a regression model, that is, a categorical variable with multiple levels, that variable should be treated as a whole. So you should mostly disregard the t-tests for ...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible