14
$\begingroup$

Lots of emphasis is placed on relying on and reporting effect sizes rather than p-values in applied research (e.g. quotes further below).

But is it not the case that an effect size just like a p-value is a random variable and as such can vary from sample to sample when the same experiment is repeated? In other words, I'm asking what statistical features (e.g., effect size is less variable from sample to sample than p-value) make effect sizes better evidence-measuring indices than p-values?

I should, however, mention an important fact that separates a p-value from an effect size. That is, an effect size is something to be estimated because it has a population parameter but a p-value is nothing to be estimated because it doesn't have any population parameter.

To me, effect size is simply a metric that in certain areas of research (e.g., human research) helps transforming empirical findings that come from various researcher-developed measurement tools into a common metric (fair to say using this metric human research can better fit the quant research club).

Maybe if we take a simple proportion as an effect size, the following (in R) is what shows the supremacy of effect sizes over p-values? (p-value changes but effect size doesn't)

binom.test(55, 100, .5)  ## p-value = 0.3682  ## proportion of success 55% 

binom.test(550, 1000, .5) ## p-value = 0.001731 ## proportion of success 55%

Note that most effect sizes are linearly related to a test statistic. Thus, it is an easy step to do null-hypothesis testing using effect sizes.

For example, t statistic resulted from a pre-post design can easily be converted to a corresponding Cohen's d effect size. As such, distribution of Cohen's d is simply the scale-location version of a t distribution.

The quotes:

Because p-values are confounded indices, in theory 100 studies with varying sample sizes and 100 different effect sizes could each have the same single p-value, and 100 studies with the same single effect size could each have 100 different values for p-value.

or

p-value is a random variable that varies from sample to sample. . . . Consequently, it is not appropriate to compare the p-values from two distinct experiments, or from tests on two variables measured in the same experiment, and declare that one is more significant than the other?

Citations:

Thompson, B. (2006). Foundations of behavioral statistics: An insight-based approach. New York, NY: Guilford Press.

Good, P. I., & Hardin, J. W. (2003). Common errors in statistics (and how to avoid them). New York: Wiley.

$\endgroup$
  • 12
    $\begingroup$ I don't draw the same conclusions from the quotations (that effect sizes are "superior" or should be reported instead of p-values). I am aware some people have overreacted by making statements like that (such as the BASP ban on p-values). It isn't a one-or-the-other situation: it's a case of pointing out that p-values and effect sizes give different kinds of useful information. Ordinarily one should not be examined without considering it in the context of the other. $\endgroup$ – whuber Aug 17 '17 at 19:28
  • 1
    $\begingroup$ I personally think reporting an estimate along with a confidence interval is enough. It gives the effect size (practical significance) and hypothesis testing (statistical significance) at the same time. $\endgroup$ – Jirapat Samranvedhya Aug 17 '17 at 20:23
  • 1
    $\begingroup$ Whether p values or effect sizes are 'superior' depends on your perspective. The former follows from the Fisherian NHST tradition, while the latter from Neyman-Pearson tradition. In some fields (biological sciences, humanities), effect sizes tend to be very small, making p values attractive. Conversely, as others note, p-values can be 'forced' smaller through changes in design, like increased N. $\endgroup$ – HEITZ Aug 17 '17 at 20:51
  • 3
    $\begingroup$ Is a screwdriver superior to a hammer? $\endgroup$ – kjetil b halvorsen Oct 27 '17 at 11:23
  • $\begingroup$ Is a nut superior to a bolt? $\endgroup$ – Martijn Weterings Feb 2 at 8:23
20
$\begingroup$

The advice to provide effect sizes rather than P-values is based on a false dichotomy and is silly. Why not present both?

Scientific conclusions should be based on a rational assessment of available evidence and theory. P-values and observed effect sizes alone or together are not enough.

Neither of the quoted passages that you supply is helpful. Of course P-values vary from experiment to experiment, the strength of evidence in the data varies from experiment to experiment. The P-value is just a numerical extraction of that evidence by way of the statistical model. Given the nature of the P-value, it is very rarely relevant to analytical purposes to compare one P-value with another, so perhaps that is what the quote author is trying to convey.

If you find yourself wanting to compare P-values then you probably should have performed a significance test on a different arrangement of the data in order to sensibly answer the question of interest. See these questions: p-values for p-values? and If one group's mean differs from zero but the other does not, can we conclude that the groups are different?

So, the answer to your question is complex. I do not find dichotomous responses to data based on either P-values or effect sizes to be useful, so are effect sizes superior to P-values? Yes, no, sometimes, maybe, and it depends on your purpose.

$\endgroup$
  • $\begingroup$ I think it would preferable to present the effect size and its confidence interval, provided the analyst is correctly able to state what a meaningful effect size is for the study at hand. The confidence interval, unlike the p-value, gives the reader a sense of both the precision of the estimate as well as its extremity. $\endgroup$ – AdamO Aug 18 '17 at 22:15
  • 1
    $\begingroup$ @AdamO Yes, I largely agree, but the P-value has two things to offer and should not be omitted. It is an index of the strength of evidence against the null, something that can only be gotten from a confidence interval by a very experienced eye, and an exact P-value does not directly invite the dichotomy of inside/outside that the confidence interval does. Of course, a likelihood function offers advantages over both. $\endgroup$ – Michael Lew Aug 19 '17 at 0:13
14
$\begingroup$

In the context of applied research, effect sizes are necessary for readers to interpret the practical significance (as opposed to statistical significance) of the findings. In general, p-values are far more sensitive to sample size than effect sizes are. If an experiment measures an effect size accurately (i.e. it is sufficiently close to the population parameter it is estimating) but yields a non-significant p-value then, all things being equal, increasing the sample size will result in the same effect size but a lower p-value. This can be demonstrated with power analyses or simulations.

In light of this, it is possible to achieve highly significant p-values for effect sizes that have no practical significance. In contrast, study designs with low power can produce non-significant p-values for effect sizes of great practical importance.

It is difficult to discuss the concepts of statistical significance vis-a-vis effect size without a specific real-world application. As an example, consider an experiment that evaluates the effect of a new studying method on students' grade point average (GPA). I would argue that an effect size of 0.01 grade points has little practical significance (i.e. 2.50 compared to 2.51). Assuming a sample size of 2,000 students in both treatment and control groups, and a population standard deviation of 0.5 grade points:

set.seed(12345)
control.data <- rnorm(n=2000, mean = 2.5, sd = 0.5)
set.seed(12345)
treatment.data <- rnorm(n=2000, mean = 2.51, sd = 0.5)
t.test(x = control.data, y = treatment.data, alternative = "two.sided", var.equal = TRUE) 

treatment sample mean = 2.51

control sample mean = 2.50

effect size = 2.51 - 2.50 = 0.01

p = 0.53

Increasing the sample size to 20,000 students and holding everything else constant yields a significant p-value:

set.seed(12345)
control.data <- rnorm(n=20000, mean = 2.5, sd = 0.5)
set.seed(12345)
treatment.data <- rnorm(n=20000, mean = 2.51, sd = 0.5)
t.test(x = control.data, y = treatment.data, alternative = "two.sided", var.equal = TRUE)  

treatment sample mean = 2.51

control sample mean = 2.50

effect size = 2.51 - 2.50 = 0.01

p = 0.044

Obviously it's no trivial thing to increase the sample size by an order of magnitude! However, I think we can all agree that the practical improvement offered by this study method is negligible. If we relied solely on the p-value then we might believe otherwise in the n=20,000 case.

Personally I advocate for reporting both p-values and effect sizes. And bonus points for t- or F-statistics, degrees of freedom and model diagnostics!

$\endgroup$
  • 2
    $\begingroup$ Darren, please show what you exactly mean in R or something just like PO. $\endgroup$ – user138773 Aug 17 '17 at 20:45
  • 7
    $\begingroup$ @Darrent James There is no practical importance in a difference between p=0.065 and p=0.043 beyond the unfortunate assumption that p=0.05 is a bright line that should be respected. Neither P-value represents compelling evidence for or against anything by itself. $\endgroup$ – Michael Lew Aug 17 '17 at 22:07
  • $\begingroup$ @Michael Lew Yes, I agree! $\endgroup$ – Darren James Aug 17 '17 at 22:13
  • 1
    $\begingroup$ James, given your code and explanations, you seem to have completely misunderstood the OP's point. Your R code also is wrong! Because you have NOt set the var.equal = TRUE while your sds are equal. With such background, I'm not sure why you even posted a response like this. OP is asking a question that doesn't have an easy answer at least at the present time! $\endgroup$ – user138773 Aug 18 '17 at 15:32
  • 1
    $\begingroup$ I've added var.equal = TRUE to the code. But it's unnecessary in this case. The same p-values are obtained with both var.equal = TRUE and the default var.equal = FALSE. $\endgroup$ – Darren James Aug 18 '17 at 19:26
5
$\begingroup$

I currently work in the data science field, and before then I worked in education research. While at each "career" I've collaborated with people who did not come from a formal background in statistics, and where emphasis of statistical (and practical) significance is heavily placed on the p-value. I've learned include and emphasize effect sizes in my analyses because there is a difference between statistical significance and practical significance.

Generally, the people I worked with cared about one thing "does our program/feature make and impact, yes or no?". To a question like this, you can do something as simple as a t-test and report to them "yes, your program/feature makes a difference". But how large or small is this "difference"?

First, before I begin delving into this topic, I'd like to summarize what we refer to when speaking of effect sizes

Effect size is simply a way of quantifying the size of the difference between two groups. [...] It is particularly valuable for quantifying the effectiveness of a particular intervention, relative to some comparison. It allows us to move beyond the simplistic, 'Does it work or not?' to the far more sophisticated, 'How well does it work in a range of contexts?' Moreover, by placing the emphasis on the most important aspect of an intervention - the size of the effect - rather than its statistical significance (which conflates effect size and sample size), it promotes a more scientific approach to the accumulation of knowledge. For these reasons, effect size is an important tool in reporting and interpreting effectiveness.

It's the Effect Size, Stupid: What effect size is and why it is important

Next, what is a p-value, and what information does it provide us? Well, a p-value, in as few words as possible, is a probability that the observed difference from the null distribution is by pure chance. We therefore reject (or fail to accept) the null hypothesis when this p-value is smaller than a threshold ($\alpha$).

Why Isn't the P Value Enough?

Statistical significance is the probability that the observed difference between two groups is due to chance. If the P value is larger than the alpha level chosen (eg, .05), any observed difference is assumed to be explained by sampling variability. With a sufficiently large sample, a statistical test will almost always demonstrate a significant difference, unless there is no effect whatsoever, that is, when the effect size is exactly zero; yet very small differences, even if significant, are often meaningless. Thus, reporting only the significant P value for an analysis is not adequate for readers to fully understand the results.

And to corroborate @DarrenJames's comments regarding large sample sizes

For example, if a sample size is 10 000, a significant P value is likely to be found even when the difference in outcomes between groups is negligible and may not justify an expensive or time-consuming intervention over another. The level of significance by itself does not predict effect size. Unlike significance tests, effect size is independent of sample size. Statistical significance, on the other hand, depends upon both sample size and effect size. For this reason, P values are considered to be confounded because of their dependence on sample size. Sometimes a statistically significant result means only that a huge sample size was used. [There is a mistaken view that this behaviour represents a bias against the null hypothesis. Why does frequentist hypothesis testing become biased towards rejecting the null hypothesis with sufficiently large samples? ]

Using Effect Size—or Why the P Value Is Not Enough

Report Both P-value and Effect Sizes

Now to answer the question, are effect sizes superior to p-values? I would argue, that these each serve as importance components in statistical analysis that cannot be compared in such terms, and should be reported together. The p-value is a statistic to indicate statistical significance (difference from the null distribution), where the effect size puts into words how much of a difference there is.

As an example, say your supervisor, Bob, who is not very stats-friendly is interested in seeing if there was a significant relationship between wt (weight) and mpg (miles per gallon). You start the analysis with hypotheses

$$ H_0: \beta_{mpg} = 0 \text{ vs } H_A: \beta_{mpg} \neq 0 $$

being tested at $\alpha = 0.05$

> data("mtcars")
> 
> fit = lm(formula = mpg ~ wt, data = mtcars)
> 
> summary(fit)

Call:
lm(formula = mpg ~ wt, data = mtcars)

Residuals:
    Min      1Q  Median      3Q     Max 
-4.5432 -2.3647 -0.1252  1.4096  6.8727 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)  37.2851     1.8776  19.858  < 2e-16 ***
wt           -5.3445     0.5591  -9.559 1.29e-10 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 3.046 on 30 degrees of freedom
Multiple R-squared:  0.7528,    Adjusted R-squared:  0.7446 
F-statistic: 91.38 on 1 and 30 DF,  p-value: 1.294e-10

From the summary output we can see that we have a t-statistic with a very small p-value. We can comfortably reject the null hypothesis and report that $\beta_{mpg} \neq 0$. However, your boss asks, well, how different is it? You can tell Bob, "well, it looks like there is a negative linear relationship between mpg and wt. Also, can be summarized that for every increased unit in wt there is a decrease of 5.3445 in mpg"

Thus, you were able to conclude that results were statistically significant, and communicate the significance in practical terms.

I hope this was useful in answering your question.

$\endgroup$
  • $\begingroup$ Jon, thanks, there are LOTS of grey areas that I was hoping to hear more about but I didn't. In lots of situations effect sizes and p-values don't agree. Many trust effect sizes in such situations which I wanted to know why. I was hoping to hear more about simulations that could show important points. Regarding the matter you brought up i.e., that effect size might be tiny but not exactly zero; methods of equivalence testing have been in place for several years now. I like the Bayesian equivalence testing even more. Anyways, I probably did n't ask my question clearly enough. -- Thanks $\endgroup$ – rnorouzian Aug 18 '17 at 18:58
  • $\begingroup$ BTW, a colleague commented that Daren's R code is wrong, it seems s/he is right. He has not put var.equal = TRUE. $\endgroup$ – rnorouzian Aug 18 '17 at 18:59
  • $\begingroup$ * In lots of situations effect sizes and p-values don't agree.* -- can you provide more information on this? An example? Regarding the matter you brought up i.e., that effect size might be tiny but not exactly zero -- this situation can result in a large sample size. Thus if the effect size is nearly zero, then the variable of interest may not impact the outcome significantly, or the relationship may be incorrectly specified (e.g. linear vs nonlinear). $\endgroup$ – Jon Aug 18 '17 at 19:06
  • $\begingroup$ Just try this tool. Also see this document. It seems I will need to ask another question at a later time using some code for clarity. -- Thank you. $\endgroup$ – rnorouzian Aug 18 '17 at 19:27
  • $\begingroup$ @rnorouzian, okay, I ran your code. What's your point? $\endgroup$ – Jon Aug 18 '17 at 19:57
4
$\begingroup$

The utility of effect sizes relative to p-values (as well as other metrics of statistical inference) is routinely debated in my field—psychology—and the debate is currently “hotter”, than normal for reasons that are relevant to your question. And though I am sure psychology isn’t necessarily the most statistically sophisticated scientific field, it has readily discussed, studied—and at times, demonstrated—limitations of various approaches to statistical inference, or at least how they are limited by human use. The answers already posted include good insights, but in case you are interested in a more extensive list (and references) of reasons for and against each, see below.

Why are p-values undesirable?

  • As Darren James notes (and his simulation shows), p-values are largely contingent on the number of observations that you have (see Kirk, 2003)
  • As Jon notes, p-values represent the conditional probability of observing data as extreme or more extreme given that the null hypothesis is true. As most researchers would rather have probabilities of the research hypothesis, and/or the null-hypothesis, p-values do not speak to probabilities in which researchers are most interested (i.e., of the null or research hypothesis, see Dienes, 2008)
  • Many who use p-values do not understand what they mean/do not mean (Schmidt & Hunter, 1997). Michael Lew’s reference to Gelman and Stern’s (2006) paper further underscores researcher misunderstandings about what one can (or cannot) interpret from p-values. And as a relatively recent story on FiveThirtyEight demonstrates, this continues to be the case.
  • p-values are not great at predicting subsequent p-values (Cumming, 2008)
  • p-values are often misreported (more often inflating significance), and misreporting is linked to an unwillingness to share data (Bakker & Wicherts, 2011; Nuijten et al., 2016; Wicherts et al., 2011)
  • p-values can be (and historically, have been) actively distorted through analytic flexibility, and are therefore untrustworthy (John et al., 2012; Simmons et al., 2011)
  • p-values are disproportionately significant, as academic systems appear to reward scientists for statistical significance over scientific accuracy (Fanelli, 2010; Nosek et al., 2012; Rosenthal, 1979)

Why are effect sizes desirable?

Note that I am interpreting your question as referring specifically to standardized effect sizes, as you say they allow researchers to transform their findings “INTO A COMMON metric”.

  • As Jon and Darren James indicate, effect sizes indicate the magnitude of an effect, independent of the number of observations (American Psychological Association 2010; Cumming, 2014) as opposed to making dichotomous decisions of whether an effect is there or not there.
  • Effect sizes are valuable because they make meta-analyses possible, and meta-analysis drive cumulative knowledge (Borenstein et al., 2009; Chan & Arvey, 2012)
  • Effect sizes help to facilitate sample size planning via a priori power analysis, and therefore efficient resource allocation in research (Cohen, 1992)

Why are p-values desirable?

Though they are less frequently espoused, p-values have a number of perks. Some are well-known and longstanding, whereas others are relatively new.

  • P-values provide a convenient and familiar index of the strength of evidence against the statistical model null hypothesis.

  • When calculated correctly, p-values provide a means of making dichotomous decisions (which are sometimes necessary), and p-values help keep long-run false-positive error rates at an acceptable level (Dienes, 2008; Sakaluk, 2016) [It is not strictly correct to say that P-values are required for dichotomous decisions. They are indeed widely used that way, but Neyman & Pearson used 'critical regions' in the test statistic space for that purpose. See this question and its answers]

  • p-values can be used to facilitate continuously efficient sample size planning (not just one-time power-analysis) (Lakens, 2014)
  • p-values can be used to facilitate meta-analysis and evaluate evidential value (Simonsohn et al., 2014a; Simonsohn et al., 2014b). See this blogpost for an accessible discussion of how distributions of p-values can be used in this fashion, as well as this CV post for a related discussion.
  • p-values can be used forensically to determine whether questionable research practices may have been used, and how replicable results might be (Schimmack, 2014; also see Schönbrodt’s app, 2015)

Why are effect sizes undesirable (or overrated)?

Perhaps the most counter-intuitive position to many; why would reporting standardized effect sizes be undesirable, or at the very least, overrated?

  • In some cases, standardized effect sizes aren’t all that they are cracked up to be (e.g., Greenland, Schlesselman, & Criqui, 1986). Baguely (2009), in particular, has a nice description of some of the reasons why raw/unstandardized effect sizes may be more desirable.
  • Despite their utility for a priori power analysis, effect sizes are not actually used reliably to facilitate efficient sample-size planning (Maxwell, 2004)
  • Even when effect sizes are used in sample size planning, because they are inflated via publication bias (Rosenthal, 1979) published effect sizes are of questionable utility for reliable sample-size planning (Simonsohn, 2013)
  • Effect size estimates can be—and have been—systemically miscalculated in statistical software (Levine & Hullet, 2002)
  • Effect sizes are mistakenly extracted (and probably misreported) which undermines the credibility of meta-analyses (Gøtzsche et al., 2007)
  • Lastly, correcting for publication bias in effect sizes remains ineffective (see Carter et al., 2017), which, if you believe publication bias exists, renders meta-analyses less impactful.

Summary

Echoing the point made by Michael Lew, p-values and effect sizes are but two pieces of statistical evidence; there are others worth considering too. But like p-values and effect sizes, other metrics of evidential value have shared and unique problems too. Researchers commonly misapply and misinterpret confidence intervals (e.g., Hoekstra et al., 2014; Morey et al., 2016), for example, and the outcome of Bayesian analyses can distorted by researchers, just like when using p-values (e.g., Simonsohn, 2014).

All metrics of evidence have won and all must have prizes.

References

American Psychological Association. (2010). Publication manual of the American Psychological Association (6th edition). Washington, DC: American Psychological Association.

Baguley, T. (2009). Standardized or simple effect size: What should be reported?. British Journal of Psychology, 100(3), 603-617.

Bakker, M., & Wicherts, J. M. (2011). The (mis) reporting of statistical results in psychology journals. Behavior research methods, 43(3), 666-678.

Borenstein, M., Hedges, L. V., Higgins, J., & Rothstein, H. R. (2009). Introduction to meta-analysis. West Sussex, UK: John Wiley & Sons, Ltd.

Carter, E. C., Schönbrodt, F. D., Gervais, W. M., & Hilgard, J. (2017, August 12). Correcting for bias in psychology: A comparison of meta-analytic methods. Retrieved from osf.io/preprints/psyarxiv/9h3nu

Chan, M. E., & Arvey, R. D. (2012). Meta-analysis and the development of knowledge. Perspectives on Psychological Science, 7(1), 79-92.

Cohen, J. (1992). A power primer. Psychological Bulletin, 112(1), 155-159. 

Cumming, G. (2008). Replication and p intervals: p values pre- dict the future only vaguely, but confidence intervals do much better. Perspectives on Psychological Science, 3, 286– 300.

Dienes, D. (2008). Understanding psychology as a science: An introduction to scientific and statistical inference. New York, NY: Palgrave MacMillan.

Fanelli, D. (2010). “Positive” results increase down the hierarchy of the sciences. PloS one, 5(4), e10068.

Gelman, A., & Stern, H. (2006). The difference between “significant” and “not significant” is not itself statistically significant. The American Statistician, 60(4), 328-331.

Gøtzsche, P. C., Hróbjartsson, A., Marić, K., & Tendal, B. (2007). Data extraction errors in meta-analyses that use standardized mean differences. JAMA, 298(4), 430-437.

Greenland, S., Schlesselman, J. J., & Criqui, M. H. (1986). The fallacy of employing standardized regression coefficients and correlations as measures of effect. American Journal of Epidemiology, 123(2), 203-208.

Hoekstra, R., Morey, R. D., Rouder, J. N., & Wagenmakers, E. J. (2014). Robust misinterpretation of confidence intervals. Psychonomic bulletin & review, 21(5), 1157-1164.

John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. PsychologicalSscience, 23(5), 524-532.

Kirk, R. E. (2003). The importance of effect magnitude. In S. F. Davis (Ed.), Handbook of research methods in experimental psychology (pp. 83–105). Malden, MA: Blackwell.

Lakens, D. (2014). Performing high‐powered studies efficiently with sequential analyses. European Journal of Social Psychology, 44(7), 701-710.

Levine, T. R., & Hullett, C. R. (2002). Eta squared, partial eta squared, and misreporting of effect size in communication research. Human Communication Research, 28(4), 612-625.

Maxwell, S. E. (2004). The persistence of underpowered studies in psychological research: causes, consequences, and remedies. Psychological methods, 9(2), 147.

Morey, R. D., Hoekstra, R., Rouder, J. N., Lee, M. D., & Wagenmakers, E. J. (2016). The fallacy of placing confidence in confidence intervals. Psychonomic bulletin & review, 23(1), 103-123.

Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7(6), 615-631.

Nuijten, M. B., Hartgerink, C. H., van Assen, M. A., Epskamp, S., & Wicherts, J. M. (2016). The prevalence of statistical reporting errors in psychology (1985–2013). Behavior research methods, 48(4), 1205-1226.

Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86(3), 638-641.

Sakaluk, J. K. (2016). Exploring small, confirming big: An alternative system to the new statistics for advancing cumulative and replicable psychological research. Journal of Experimental Social Psychology, 66, 47-54.

Schimmack, U. (2014). Quantifying Statistical Research Integrity: The Replicability-Index. Retrieved from http://www.r-index.org 

Schmidt, F. L., & Hunter, J. E. (1997). Eight common but false objections to the discontinuation of significance testing in the analysis of research data. In L. L. Harlow, S. A. Mulaik, & J. H. Steiger (Eds.), What if there were no significance tests? (pp. 37–64). Mahwah, NJ: Erlbaum.

Schönbrodt, F. D. (2015). p-checker: One-for-all p-value analyzer. Retrieved from http://shinyapps.org/apps/p-checker/

Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological science, 22(11), 1359-1366.

Simonsohn, U. (2013). The folly of powering replications based on observed effect size. Retreived from http://datacolada.org/4

Simonsohn, U. (2014). Posterior-hacking. Retrieved from http://datacolada.org/13.

Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014). P-curve: A key to the file-drawer. Journal of Experimental Psychology: General, 143(2), 534-547.

Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014). P-curve and effect size: Correcting for publication bias using only significant results. Perspectives on Psychological Science, 9(6), 666-681.

Wicherts, J. M., Bakker, M., & Molenaar, D. (2011). Willingness to share research data is related to the strength of the evidence and the quality of reporting of statistical results. PloS one, 6(11), e26828.

$\endgroup$
  • 2
    $\begingroup$ Very nice collection of ideas and references. It should be helpful for those who want to dig a bit further, but note that many of the points have relevant questions and answers on this site. Links to those would help too. $\endgroup$ – Michael Lew Aug 19 '17 at 0:35
  • $\begingroup$ @MichaelLew Thanks. I'll see about adding some links when I have the time later--it took me the better part of the afternoon to draft this response, and assemble the references. Regarding your edit, I think your point is well-taken, but perhaps more of an addition, as opposed to a correction? I said p-values provide a means of making dichotomous decisions (not that they are "required", or the only way of doing so). I agree that N-P critical regions are another way, but I responded to the OP in context of what p-values afford vs. standardized effect sizes. $\endgroup$ – jsakaluk Aug 19 '17 at 2:30
  • 1
    $\begingroup$ jsakaluk, yes I can see that you would have spent a long time on the answer and it is very useful and worthy of your effort. I edited the item on advantages of P-values because you wrote "When used correctly" they can be dichotomised, whereas the reality is that such a use ignores much of the information that is encoded in the P-value and so is arguably (and in my opinion) an incorrect use. I did not want to subvert your intention and so I changed "used" to "calculated". $\endgroup$ – Michael Lew Aug 19 '17 at 21:25
3
$\begingroup$

From the perspective of an Epidemiologist, on why I prefer effect sizes over p-values (though as some people have noted, it's something of a false dichotomy):

  1. The effect size tells me what I actually want, the p-value just tells me if it's distinguishable from null. A relative risk of 1.0001, 1.5, 5, and 50 might all have the same p-value associated with them, but mean vastly different things in terms of what we might need to do at a population level.
  2. Relying on a p-value reinforces the notion that significance-based hypothesis testing is the end-all, be-all of evidence. Consider the following two statements: "Doctors smiling at patients was not significantly associated with an adverse outcome during their hospital stay." vs. "Patients who had their doctor smile at them were 50% less likely to have an adverse outcome (p = 0.086)." Would you still, maybe, given it has absolutely no cost, consider suggesting doctors smile at their patients?
  3. I work with a lot of stochastic simulation models, wherein sample size is a function of computing power and patience, and p-values are essentially meaningless. I have managed to get p < 0.05 results for things that have absolutely no clinical or public health relevance.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.