23

The first sentence of the current 2015 editorial to which the OP links, reads: The Basic and Applied Social Psychology (BASP) 2014 Editorial *emphasized* that the null hypothesis significance testing procedure (NHSTP) is invalid... (my emphasis) In other words, for the editors it is an already proven scientific fact that "null hypothesis ...


21

The basic issue here is true and fairly well known in statistics. However, his interpretation / claim is extreme. There are several issues to be discussed: First, power doesn't change very fast with changes in $N$. (Specifically, it changes as a function of $\sqrt N$, so to halve the standard deviation of your sampling distribution, you need to ...


21

The advice to provide effect sizes rather than P-values is based on a false dichotomy and is silly. Why not present both? Scientific conclusions should be based on a rational assessment of available evidence and theory. P-values and observed effect sizes alone or together are not enough. Neither of the quoted passages that you supply is helpful. Of course ...


20

Many people would argue that a $p$-value can either be significant ($p< \alpha$) or not, and so it does not (ever) make sense to compare two $p$-values between each other. This is wrong; in some cases it does. In your particular case there is absolutely no doubt that you can directly compare the $p$-values. If the sample size is fixed ($n=1000$), then $p$...


20

That is one measure of effect size, but there are many others. It is certainly not the $t$ test statistic. Your measure of effect size is often called Cohen's $d$ (strictly speaking that is correct only if the SD is estimated via MLE—i.e., without Bessel's correction); more generically, it is called the 'standardized mean difference'. Perhaps this ...


19

Are smaller $p$-values "more convincing"? Yes, of course they are. In the Fisher framework, $p$-value is a quantification of the amount of evidence against the null hypothesis. The evidence can be more or less convincing; the smaller the $p$-value, the more convincing it is. Note that in any given experiment with fixed sample size $n$, the $p$-value is ...


19

I feel that banning hypothesis tests is a great idea except for a select few "existence" hypotheses, e.g. testing the null hypothesis that there is not extra-sensory perception where all one would need to demonstrate to have evidence that ESP exists is non-randomness. But I think the journal missed the point that the main driver of poor research in ...


18

Effect sizes Common standardised effect sizes typically quantify the amount or degree of a relationship or effect. The most common effect size measures are probably cohen's d, Pearson's r, and the odds ratio (particularly for a binary predictor). Less common effect size measures: That said, you can have standardised and unstandardised effect size measures. ...


16

No, a sample cannot be too large for an ANOVA or a t-test. You will almost invariably get statistically significant results because you have a great deal of power; however, this does not mean that you detect differences that are false. Indeed, regardless of how many cases you have, an effect that does not exist will not become significant. This is a common ...


16

The delta method simply says that if you can represent an auxiliary variable you can represent as a function of normally distributed random variables, that auxiliary variable is approximately normally distributed with variance corresponding to how much the auxiliary varies with respect to the normal variables (EDIT: as pointed out by Alecos Papadopoulos the ...


15

I expect someone with a background in a more relevant area (psychology or education, say) will chime in with a better answer, but I'll give it a shot. "Effect size" is a term with more than one meaning -- which many years past led some some confused conversations until I eventually came to that realization. Here we're clearly dealing with the scaled-for-...


14

It rather depends on what you mean by "by hand". There is more than one way to do it. You can use the residuals: > etasq(xyaov) Partial eta^2 x 0.4854899 Residuals NA > 1 - var(xyaov$residuals)/var(y) [1] 0.4854899 (You didn't set a seed, so we don't have exactly the same result). Almost equivalently, you can use ...


14

In the context of applied research, effect sizes are necessary for readers to interpret the practical significance (as opposed to statistical significance) of the findings. In general, p-values are far more sensitive to sample size than effect sizes are. If an experiment measures an effect size accurately (i.e. it is sufficiently close to the population ...


13

The estimator that corresponds to the Wilcoxon test is the Hodges-Lehmann estimator; it's returned by wilcox.test using the conf.int=TRUE option, under "difference in location". For your example: > wilcox.test(b~a,data=d, conf.int=TRUE) Wilcoxon rank sum test data: b by a W = 355, p-value = 6.914e-06 alternative hypothesis: true location shift ...


13

Answering this (good) question responsibly probably requires addressing meta-analysis topics beyond conventional meta-regression. I've encountered this issue in consulting clients' meta-analyses but haven't yet found or developed a satisfactory solution, so this answer isn't definitive. Below I mention five relevant ideas with selected reference citations. ...


13

I see this approach as an attempt to address the inability of social psychology to replicate many previously published 'significant findings.' Its disadvantages are: that it doesn't address many of the factors leading to spurious effects. E.g., A) People can still peek at their data and stop running their studies when an effect size strikes them as ...


12

There is a fundamental confusion here: The $p$-value you got comes from the $\chi^2$ test. It tells you the probability of getting a $\chi^2$ statistic as extreme or more extreme than yours if the null hypothesis is true. It tells you nothing about how big the effect is. On the other hand, Cramer's $V$ is a measure of effect size. It tells you how ...


11

We know that: $$ F = \frac{MS_B} {MS_W} = \frac{SS_B/(k-1)} {SS_W/(N-k)}. $$ Thus $SS_B = F \times MS_W \times (k-1)$, and $SS_W = MS_W \times (N-k)$. We also know that: $$ \eta^2 = \frac{SS_B}{SS_B + SS_W} $$ Thus, if we substitute (1) in (2): $$ \eta^2 = \frac{F \times MS_W \times (k-1)}{F \times MS_W \times (k-1) + MS_W \times (N-k)} $$ The $MS_W$ ...


11

Built on Adam's answers, I have a few elaborations. First and most important, it is not easy to conceptualize substantive theories on how and why one effect size predicts another effect size. A multivariate meta-analysis is usually sufficient to explain the association among the effect sizes. If you are interested in hypothesizing directions among the effect ...


11

You are right, the difference between them is very small and with large $N$ will disappear. In fact most people (at least in my experience) are not aware of any of this; "Cohen's $d$" is often used generically, many people have not heard of Hedges' $g$, but they use the latter formula and call it by the former name. The difference is that Cohen used the ...


11

eta-squared ($\eta^2$), is a measure of effect size for ANOVA models that is analogous to $R^2$. That is, it gives the proportion of the variability in $Y$ that can be accounted for by knowledge of $X$. There is a 'regular' $\eta^2$, and a partial $\eta^2$. This distinction only comes into play when you have an ANOVA with multiple factors. Here are the ...


11

the danger occurs that "everything becomes significant" (even minor, practically irrelevant effects). This is not an argument against large sample sizes, it's a direct argument against hypothesis testing for your particular problem. If you have a problem rejecting for small effect sizes don't use ordinary hypothesis tests. It may be that you need an ...


10

The answer is almost always: report both. This way, your audience can decide on the interestingness and importance of your results, instead of just having to believe you. Confidence intervals are similarly always useful, because they give a neat indication of both effect size, and significance, in one. Even better if it's on a graph :) It may be best to ...


10

Hi Erica and welcome to the site. Have a look at this (page 3) document and this paper. The basic formula for the conversion is $$ d=\mathrm{LogOR}\times \frac{\sqrt{3}}{\pi} $$ Applying the delta-method, we get the following expression for the the variance of $d$ (the standard error of $d$ is just the square root of its variance): $$ \mathrm{Var}_{d}=\...


10

There is a growing opinion among statisticians that Cohen's $d$ has more problems than advantages. I recommend that you compute effect estimates in raw data or subject-matter units. Besides losing subject-matter interpretability, Cohen's $d$ invites one to make arbitrary categorizations as you did. See http://biostat.mc.vanderbilt.edu/ManuscriptChecklist ...


9

As you point out, there are merits with all three approaches. There clearly isn't one option that is 'best'. Why not do all 3 and present the results as a sensitivity analysis? A meta-analysis conducted with ample and appropriate sensitivity analyses just shows that the author is well aware of the limits of the data at hand, makes explicit the influence of ...


9

One choice of effect size for the Mann-Whitney U test is the common language effect size. For the Mann-Whitney U, this is the proportion of sample pairs that supports a stated hypothesis. A second choice is the rank correlation; because the rank correlation ranges from -1 to +1, it has properties that are similar to the Pearson r. In addition, by the ...


9

Questions: How come an irrelevant regressor turn out statistically significant? I think it's helpful to think about what happens as your sample size approaches the population itself. Significance testing is meant to give you an idea of whether not an effect exists in the population. This is the reason why when working with census data (that surveys the ...


9

I don't know what's meant by smaller p-values being "better", or by us being "more confident in" them. But regarding p-values as a measure of how surprised we should be by the data, if we believed the null hypothesis, seems reasonable enough; the p-value is a monotonic function of the test statistic you've chosen to measure discrepancy with the null ...


Only top voted, non community-wiki answers of a minimum length are eligible