Questions tagged [effect-size]
Effect size is "a measure of the strength of a phenomenon or a sample-based estimate of that quantity" [Wikipedia].
962
questions
74
votes
4
answers
6k
views
A psychology journal banned p-values and confidence intervals; is it indeed wise to stop using them?
On 25 February 2015, the journal Basic and Applied Social Psychology issued an editorial banning $p$-values and confidence intervals from all future papers.
Specifically, they say (formatting and ...
51
votes
1
answer
422k
views
How to interpret and report eta squared / partial eta squared in statistically significant and non-significant analyses?
I have data that has eta squared values and partial eta squared values calculated as a measure of effect size for group mean differences.
What is the difference between eta squared and partial eta ...
46
votes
4
answers
17k
views
Are smaller p-values more convincing?
I've been reading up on $p$-values, type 1 error rates, significance levels, power calculations, effect sizes and the Fisher vs Neyman-Pearson debate. This has left me feeling a bit overwhelmed. I ...
34
votes
7
answers
96k
views
Difference between Cohen's d and Hedges' g for effect size metrics
For an effect size analysis, I am noticing that there are differences between Cohen's d, Hedges's g and Hedges' g*.
Are these three metrics normally very similar?
What would be a case where they ...
26
votes
4
answers
30k
views
What sense does it make to compare p-values to each other?
I have two populations (men and women), each containing $1000$ samples. For each sample I have two properties A & B (first year grade point average, and SAT score). I have used a t-test separately ...
25
votes
1
answer
13k
views
How to use delta method for standard errors of marginal effects?
I am interested in better understanding the delta method for approximating the standard errors of the average marginal effects of a regression model that includes an interaction term. I've looked at ...
25
votes
1
answer
4k
views
Is a priori power analysis essentially useless?
I attended a meeting of the Society for Personality and Social Psychology last week where I saw a talk by Uri Simonsohn with the premise that using an a priori power analysis to determine sample size ...
22
votes
5
answers
3k
views
What is effect size... and why is it even useful?
I have an introductory-graduate-level statistics background (assume I know mathematical statistics and probability at an undergraduate level (e.g., Wackerly et al., Ross' Probability), and have some ...
21
votes
2
answers
55k
views
Effect size to Wilcoxon signed rank test?
Some authors (e.g. Pallant, 2007, p. 225; see image below) suggest to calculate the effect size for a Wilcoxon signed rank test by dividing the test statistic by the square root of the number of ...
17
votes
4
answers
14k
views
What does it mean for a study to be over-powered?
What does it mean for a study to be over-powered?
My impression is that it means that your sample sizes are so large that you have the power to detect minuscule effect sizes. These effect sizes are ...
16
votes
2
answers
34k
views
Mann-Whitney U-test: confidence interval for effect size
According to Fritz, Morris, and Richler (2011; see below), $r$ can be calculated as an effect size for the Mann-Whitney U-test using the formula
$$
r = \frac{z}{\sqrt N}
$$
This is convenient to me, ...
15
votes
5
answers
8k
views
Are effect sizes really superior to p-values?
Lots of emphasis is placed on relying on and reporting effect sizes rather than p-values in applied research (e.g. quotes further below).
But is it not the case that an effect size just like a p-...
15
votes
3
answers
25k
views
How can I derive effect sizes in lme4 and describe the magnitude of fixed effects?
I have run a mixed effects model with a ln transformed continuous response (seconds) and found a significant effect of the categorical predictor (treatment/control, the only fixed effect in the model)....
15
votes
1
answer
1k
views
If we shouldn't do post hoc power calculations, are post hoc effect size calculations also invalid?
We have posts on here discussing why post hoc power calculations are flawed.
What about post hoc calculations of the effect size that we could catch? That seems logically equivalent. If we are not ...
14
votes
1
answer
493
views
Specifying prior for effect size in meta-analysis
My question concerns priors on effect sizes, in my project the measure is Cohen's $D$. Through reading the literature, it seems vague priors are often used, such as in the well-know eight schools ...
13
votes
6
answers
2k
views
Power paradox: overestimated effect size in low-powered study, but the estimator is unbiased
If we have an underpowered study but manage to reject the null hypothesis, anyway, it makes sense to wonder if we have overestimated the effect size.
However, such a concern seems unwarranted if we ...
13
votes
1
answer
9k
views
Variance of Cohen's $d$ statistic
Cohen’s $d$ is one of the most common ways we measure the size of an effect (see Wikipedia). It simply measures the distance between two means in terms of the pooled standard deviation. How can we ...
13
votes
3
answers
303
views
Justification for reporting non-significant effect sizes
We recently received peer review on a manuscript that contained reasonable feedback but also this item:
when the p-value is larger than the alpha level chosen (here is .05), it is not necessary to ...
13
votes
2
answers
1k
views
Understanding Gelman & Carlin "Beyond Power Calculations: ..." (2014)
I am reading Gelman & Carlin "Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors" (2014). I am trying to understand the main idea, the main takeway, but I am confused....
13
votes
1
answer
980
views
Do likelihood ratios and Bayesian model comparison provide superior & sufficient alternatives to null-hypothesis testing?
In response to a growing body of statisticians and researchers that criticize the utility of null-hypothesis testing (NHT) for science as a cumulative endeavour, the American Psychological Association ...
12
votes
2
answers
4k
views
What descriptive statistics are not effect sizes?
Wikipedia says
effect size is a measure of the strength of a phenomenon or a sample-based estimate of that quantity. An effect size calculated from data is a descriptive statistic that conveys ...
12
votes
2
answers
2k
views
Can I include an effect size as an independent variable in a meta-regression?
My question is whether I can use an effect size $X$ as a dependent variable and another effect size $Y$ as the independent variable in a meta-regression?
For example, I conducted a meta-analysis for ...
12
votes
2
answers
4k
views
Effect size for interaction effect in pre-post treatment-control design
If you choose to analyse a pre-post treatment-control design with a continuous dependent variable using a mixed ANOVA, there are various ways of quantifying the effect of being in the treatment group.
...
12
votes
2
answers
502
views
Is there a general definition of the effect size?
The effect-size tag has no wiki. The wikipedia page about the effect size does not provide a precise general definition. And I have never seen a general definition ...
12
votes
2
answers
2k
views
Bayesian analysis of contingency tables: How to describe effect size
I'm working through the examples in Kruschke's Doing Bayesian Data Analysis, specifically the Poisson exponential ANOVA in ch. 22, which he presents as an alternative to frequentist chi-square tests ...
12
votes
1
answer
2k
views
Is the Shapiro Wilk test W an effect size?
I want to avoid misusing normality tests where a large enough sample size will highlight any slight non-normality. I want to be able to say that a distribution is "normal enough".
When the population ...
11
votes
5
answers
28k
views
Calculating eta squared from F and df
I'm trying to compute ANOVA effect sizes from papers that provide an F value without other information. If I understand correctly, the effect size for a single-factor ANOVA is
$$
\eta {2} = \frac{ss_{...
11
votes
2
answers
10k
views
Effect size in GLMM
In the output of a GLMM, using a binary variable as response variable and continuous variables as explanatory variables [family = binomial(link="logit")], I obtain, for each variable, an estimate ...
11
votes
5
answers
14k
views
Omega squared for measure of effect in R?
The statistics book I am reading recommends omega squared to measure the effects of my experiments. I have already proven using a split plot design (mix of within-subjects and between-subjects design) ...
11
votes
2
answers
15k
views
Calculating effect size for variables in a multiple regression in R
I've been trying to learn about effect size in relation to linear regression and am wondering how to implement it in R. Sure, I have p-values and they indicate how "predictive" the explanatory ...
10
votes
5
answers
5k
views
Practical significance, especially with percents: "standard" measure and threshold
I, like many people, dislike statistical significance testing. I would much rather measure "practical significance" / effect size.
The problem is that I do not know of a "standard" way of doing so. ...
10
votes
2
answers
10k
views
Can a sample be too large for ANOVA or a t-test?
I have close to a million data sets and whenever I run mean comparison test, either ANOVA or a t-test, I get a significance level of less than .0001 on SPSS. I'm concerned that my sample is so large ...
10
votes
2
answers
3k
views
In a meta-analysis, how should one handle non-significant studies containing no raw data?
Let's say that I'm conducting a meta-analysis, looking at the performance of group A and group B with respect to a certain construct. Now, some of the studies that I'll come across will report that no ...
10
votes
3
answers
20k
views
Cohen's d for dependent sample t-test
Quick question: I've seen Cohen's d calculated two different ways for a dependent samples t-test (e.g., within-samples design testing the efficacy of a medication with pre/post timepoints).
Using the ...
10
votes
2
answers
16k
views
Effect size for a one-sample t-test
Imagine I work in a screw factory (which is not true). We have been producing a series of 5cm-long (on average) screws for years. I want to modify the production parameters in order to save some money ...
10
votes
2
answers
526
views
Calculating CIs for $\eta^2$ via Z scores - sample size?
In the thread
Confidence Interval for $\eta^2$ it was proposed that if only limited statistics are available (in my case, F, df1, df2, means), one could calculate the 95% CI for $\eta^2$ by:
...
10
votes
2
answers
740
views
How to best handle subscores in a meta-analysis?
I am conducting a meta-analysis of effect sizes d in R using the metafor package. d represents differences in memory scores between patients and healthy. However some studies report only subscores of ...
10
votes
3
answers
4k
views
Is the binomial effect size display (BESD) a misleading representation of effect size?
It's hard for me to accept that Donald Rubin would ever come up with a true lemon of a technique. Yet, that's my perception of the BESD [1, 2, 3].
The original paper by Rosenthal and Rubin (1982) ...
9
votes
3
answers
6k
views
Understanding $\chi^{2}$ and Cramér's $V$ results
I have two groups of buyers, A and B, and I want to test whether the difference between the percentage of them who would buy a ...
9
votes
2
answers
2k
views
Desired effect size vs. expected effect size
When estimating the effect size for determining optimal sample size for a statistical test, the textbook approach seems to do this from pilot studies or the like. Is it also legal to define effect ...
9
votes
1
answer
989
views
Consequences of exceeding sample size after determination of sample in power analysis
Challenge
In our office, I have been involved in a discussion about sample size and its influence on effect size - can you help me out and explain further?
Base
When conducting a power analysis, ...
9
votes
2
answers
8k
views
Do Cohen's d and Hedges' g apply to the Welch t-test?
How do you go about calculating effect size for the Welch t-test?
9
votes
3
answers
5k
views
Calculating effect sizes and standard errors for the difference between two standardized mean differences
I have two related questions, both of which are related to a meta-analysis I am performing where where the primary outcomes are expressed in terms of the standardized mean difference.
My studies have ...
9
votes
2
answers
15k
views
Calculating and interpreting effect sizes for interaction terms
Can an effect size be calculated for an interaction effect in general and more specifically using the F-statistic and its associated degrees of freedom? If yes, should this be done and what is the ...
9
votes
3
answers
12k
views
Linear regression effect sizes when using transformed variables
When performing linear regression, it is often useful to do a transformation such as log-transformation for the dependent variable to achieve better normal distribution conformation. Often it is also ...
9
votes
1
answer
2k
views
Why is usually the acceptable probability of type 1 and type 2 errors different?
This question is raised by my supervisor and I don't know how to explain.
Usually the accepted confident level is 0.95 which means that the probability of type 1 error is 5%. But the usually ...
9
votes
3
answers
28k
views
Reporting coefficient of determination using Spearman's rho
I have two non-normally distributed variables (positively-skewed, exhibiting ceiling effects). I would like to calculate the correlation coefficient between these two variables. Due to the non-...
9
votes
3
answers
1k
views
Teaching students about non-significant results and large effect size
This year I am going to teach statistics to sophomore year students of psychology. We'll be training such methods as one-way ANOVA. The example will be the time-reaction of a cognitive task among ...
9
votes
3
answers
3k
views
Intraclass correlation and aggregation
Imagine that:
You have a sample of 1000 teams each with 10 members.
You measured team functioning by asking each team member how well they think their team is functioning using a reliable multi-item ...
9
votes
1
answer
1k
views
Does Greenhouse-Geisser correction influence the effect size estimation, the statistical significance treshold, or both of them?
I discussed recently with one of my colleagues about the Greenhouse-Geisser correction for non-sphericity. I told him first that this correction is very conservative and that it would be more ...