7

I know I am several months late, but just want to respond to the other answers. All answers use simulations and/or claim the exact Fisher calculation is too computationally intensive. If you code this efficiently, you can get an exact computation very quickly. Below is a comparison time of the sample code fisherpower() function vs. the power.exact.test() ...


6

The problem is in the use of the "post-hoc effect size," not that its calculation is invalid. A "post-hoc effect size" is fundamentally an estimate of population parameters (e.g., mean difference between two groups and a standard deviation, not a standard error!) whose precisions might be affected by study design but aren't otherwise ...


5

The correlation coefficient in the qqplot can be used as a test for normality (in the case of a normal qqplot, or for some other null distribution model). See for instance this paper Developing a Test of Normality in the Classroom. But if the correlation in the plot is useful as an effect size, is another matter ... test statistics by themselves are not ...


4

Short answer: You don't. Since your effect is not significant (you fail to reject the null hypothesis that there is no effect), if you're following the rules of null hypothesis significance testing, you cannot conclude that there is any effect here. Additional points You're running 5 tests, without adjusting for multiple comparisons, so the chances of ...


4

The paper suggested by @simone, Brysbaert and Stevens as the title indicates, is focused on 'Power Analysis and Effect Size in Mixed Effects Models', but it includes a calculation of effect size, which is not present in @simone's answer, with a reference to Westfall et al. (2014), for the effect size calculation: '*First, Westfall et al. (2014) showed how ...


4

Inverse-variance weighting makes sense for combining parameter estimates that are in the same units. The problem is that linear and logistic regression coefficients are in different types of units. Per unit change in the same predictor, a linear regression coefficient represents the associated change in a numeric outcome, while a logistic regression ...


4

You intuition is correct here --- although the p-value is not used as a measure of effect size, you are correct that in some tests, for a fixed sample size the distribution of the p-value is monotonically related to the effect size, and thus is implicitly a transformed estimator of the effect size. Generally speaking, a larger effect size (further from the ...


3

It might be possible to obtain the precise value of standard error from other reported characteristics. For example, if the effect sizes are computed as standardized measures, such as Cohen's d or correlation coefficient, the sample size and effect size are all the information needed for obtaining the standard error. See this question for obtaining the ...


3

This is a kind of regression toward the mean applied, in this specific case, to the variance or standard deviation. Regression toward the mean is observed when selecting subjects based on a very high of very low value and observing that subsequent measurements will be closer to the average. Regression toward the mean can be observed, for instance, if you ...


3

There are many types of effect sizes, which depend on the nature of the study outcome, the study design, the research question and the statistical test or model used to answer this question. For example, if a study compares two groups - an experimental group and a control group - with respect to a continuous outcome value (say, blood pressure) and the two ...


3

The pwr.2p2n.test function is based on the testing of proportions with Cohen's h and the variance stabilizing transformation (See for the original source of this statistic: Jacob Cohen 1966) $$\Phi = 2 \text{arcsin} \sqrt{p}$$ These $\Phi$ are approximately normal distributed with variance $\frac{1}{N}$ For the difference between two of these transformed ...


3

Maybe the following reasoning can help you understand why 0.99 seems a suspiciously high power. An $h = 0.5$ is about the difference between the probability of success 0.7 vs 0.46 (ES.h(0.7, 0.46) = 0.49). With a sample size of 153 in each group, this is the difference between 107 and 70 successes which is quite noticeable especially since $\alpha = 0.05$ is ...


3

Let’s do two t-test examples. In the first situation, we take $25$ observations and get a mean of $0.59218$ and variance of $1.891$. Running through the one-sample t-test calculations, we get a t-stat of $2.1532$ and a p-value of $0.04157$, significant at the legendary $0.05$-level. In the second situation, we take $250,000$ observations and get a mean of $0....


3

So there are two types of studies: The variable that is considered the 'exposure' (child maltreatment) is dichotomized to create two groups (exposed vs not exposed) for which the mean levels of the 'outcome' variable (depression level) are reported and based on this a standardized mean difference can be computed. The variable that is considered the '...


3

The statistic Cohen's d follows a scaled non-central t-distribution. This statistic is the difference of the mean divided by an estimate of the sample standard deviation of the data: $$d = \frac{\bar{x}_1-\bar{x}_2}{\hat{\sigma}}$$ It is used in power analysis and relates to the t-statistic (which is used in significance testing) $$d = n^{-0.5} t $$ This ...


3

It's not a valid way to do it. Among other things, $x_1$ and $x_2$ can be correlated. Here is a simple simulation (coded in R): set.seed(9684) # makes this perfectly reproducible x1 = c(rnorm(20), rnorm(20, mean=1)) x2 = rep(0:1, each=20) cor(x1, x2) # [1] 0.4715828 these are ...


3

I think this is done with the delta method. Let $g(Z)$ be some transformation of a random variable. Then $\operatorname{Var}(g(Z)) = \operatorname{Var}(Z)[g'(Z)]^2$. In this case, $g(Z) = \tanh(Z)$, so some calculus can help us get to where we need to go. $$\operatorname{Var}(g(Z)) = 0.0103 \times [1 - \tanh^2(0.5493)]^2 \approx 0.005793821$$


3

Another option than using the delta method is to make use of the fact that the sampling variance of Fisher's $Z$ values is $1/(N-3)$ where $N$ is the sample size. The variance of $Z$ ($Var(Z)$) is reported, so the $N$ can be computed with $$ N = \frac{1}{Var(Z)}+3. $$ $N$ can then be used to compute the sampling variance of the correlation using the well-...


3

"Accounting for the variation" usually means "after regressing on the other explanatory variables and extracting the residuals" -- that is, the "left over" variation not "accounted for" by those variables. The residuals have the same structure as the original responses, so just about any graphical method of plotting ...


3

As whuber comments, in general the answer is yes. Here are the p values in a simulation of 100 experiments each with 25 participants in each one of two groups, a default t test and effect sizes 0.1, 0.5 and 0.8: The standard deviations of the p values for the three effect sizes are 0.29, 0.24 and 0.07, respectively. R code: n_sim <- 1e2 ...


3

Your setting describes a pretest-posttest-control group design, hence you should not use a (standardized) mean difference measure but a (standardized) measure of mean change. You can read a broad overview in Chapter 6 of the Cochrane Handbook and in Viechtbauer's documentation on conducting meta-analysis in R with the metafor package. What is mean change? ...


2

Since Glass' $d$ is almost identical in form to Cohen's $d$ differing only in whether to use a pooled standard deviation or the one from the control group then the same cut-off points would be appropriate although as you point out they are quite arbitrary. It would be better to compare with values typical for the particular field of science and for measures ...


2

It sounds like you are replying to your own question already. The effect size statistic is independent of your statistical significance. Here a extreme example you can run in R: data <- matrix(c(1,0,2,0,1.5,0, 1.7,1,1.8,1,2.2,1), nrow= 6, ncol = 2, byrow = TRUE) data <- as.data.frame(data) data[,2] <- as.factor(data[,2]) effsize::...


2

The effect estimate is the difference in the result from when your factor changes from the high value to the low value. In your problem statement above the effect estimate for factor A is the average etch rate when factor A=1 minus the average etch rate when A=-1. When A= 1: Average( 669, 650, 642, 635, 749, 868, 729, 860)= 725.25 When A= -1: Average( 550, ...


2

The dv, transfer, measures pretest to posttest improvement for a given level of distance If this is a continuous variable then odds ratios don't make sense and you can just use the regression coefficient as an effect size. If it it binary, then you should fit a logistic model and report odds ratios.


2

This is a great question and one I wish was asked more by research applying propensity/matching methods. In this case, there is a tradeoff between bias due to imbalance (i.e., not discarding treated units but not finding good matches for them) and bias due to distortion of the target population (i.e., discarding treated units to find good matches), or what ...


2

Generally, the p-value can be less than 0.05 even if the observed value of the estimated parameter is much smaller than the postulated value. There is a simple formula for the smallest observed value you need to have significant result. Of course, that assumes all your assumptions are correct and that the estimate is normally distributed. If you are doing ...


2

The exact conversion of a point-biserial correlation coefficient (i.e., the correlation between a binary and a numeric/quantitative variable) to a Cohen's d value is: $$d = \frac{r \sqrt{h}}{\sqrt{1-r^2}},$$ where $h = m/n_0 + m/n_1$, $m = n_0 + n_1 - 2$, and $n_0$ and $n_1$ are the number of 0's and 1's respectively. Here is some R code to demonstrate that ...


1

A chi-square test is different from your t-test or regression, you are looking for deviations from the expected counts. If you check the wiki : Pearson's chi-squared test is used to determine whether there is a statistically significant difference between the expected frequencies and the observed frequencies in one or more categories of a contingency table. ...


1

A useful reference is "Statistical power analysis for the behavioral sciences" by Jacob Cohen. Chapter 8, about ANOVA, is available via google https://books.google.ch/books?id=rEe0BQAAQBAJ&&pg=PA273 or other online sources. The value for Cohen's f resembles Cohen's d Cohen's d is the ratio of the difference between two population means and ...


Only top voted, non community-wiki answers of a minimum length are eligible