11
$\begingroup$

I am reading Gelman & Carlin "Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors" (2014). I am trying to understand the main idea, the main takeway, but I am confused. Could anyone help distill me the essence?

The paper goes something like this (if I understood it correctly).

  • Statistical studies in psychology are often plagued by small samples.
  • Conditional on a statistically significant result in a given study,
    (1) the true effect size is likely to be severely overestimated and
    (2) the sign of the effect may be opposite with high probability -- unless the sample size is large enough.
  • The above is shown using a prior guess of the effect size in population, and that effect is typically taken to be small.

My first problem is, why condition on the statistically significant result? Is it to reflect the publication bias? But that does not seem to be the case. So why, then?

My second problem is, if I do a study myself, should I treat my results differently than I am used to (I do frequentist statistics, not very familiar with Bayesian)? E.g. I would take a data sample, estimate a model and record a point estimate for some effect of interest and a confidence bound around it. Should I now mistrust my result? Or should I mistrust it if it is statistically significant? How does any given prior change that?

What is the main takeaway (1) for a "producer" of statistical research and (2) for a reader of applied statistical papers?

References:

P.S. I think the new element for me here is the inclusion of prior information, which I am not sure how to treat (coming from the frequentist paradigm).

$\endgroup$
  • $\begingroup$ As you can see, I am pretty confused, so my questions may not seem coherent or sensible. I will appreciate any hints for making more sense out of the paper I am studying. I hope to be able to pose more sensible questions as my understanding of the issue progresses. $\endgroup$ – Richard Hardy Oct 31 '16 at 16:00
  • 7
    $\begingroup$ Note that they set the premise of the paper up right at the start: "You have just finished running an experiment. You analyze the results, and you find a significant effect. Success! But wait—how much information does your study really give you? How much should you trust your results?" --- they're describing what happens/what is implied when you have significance. They use those consequences to motivate focusing on things other than significance. $\endgroup$ – Glen_b -Reinstate Monica Oct 31 '16 at 16:08
  • $\begingroup$ You should mistrust your result -- yes -- if you run multiple significance tests and filter out all that turn out to be insignificant; this is kind of a "publication bias" but it can happen without any publications, simply inside one person's lab over a course of several months' or years' of experiments. Everybody does something like that to a certain extent, hence the pedagogic interest in conditioning on significant results. $\endgroup$ – amoeba says Reinstate Monica Oct 31 '16 at 17:10
  • $\begingroup$ @amoeba, OK, but if (hypothetically) I estimate only one model and focus on only one prespecified parameter (so absolutely no multiple testing), would Gelman & Carlin's result change anything? How about including the prior information? $\endgroup$ – Richard Hardy Oct 31 '16 at 17:11
  • 2
    $\begingroup$ Prior information is needed to asses false discovery rate; the usual logic of significance testing only guarantees type I error rate P(signif | null). To estimate P(null | signif) you need to invoke some prior. That's what Gelman & Carlin are doing here. If you only estimate one model then "false discovery rate" is meaningless (in the frequentist approach); but usually people estimate many models :-) or at least they read literature that consists of other people estimating many models. $\endgroup$ – amoeba says Reinstate Monica Oct 31 '16 at 17:14
5
$\begingroup$

I re-read the paper and this time it seems much clearer. Now also the helpful comments by @Glen_b and @amoeba make lots of sense.

The whole discussion is based on a starting point that a statistically significant result has been obtained. Conditional on that, we have the estimated effect size distributed differently than it would be absent the conditioning: $$ P_{\hat\beta}(\cdot|\hat\beta \text{ is statistically significant})\neq P_{\hat\beta}(\cdot). $$ The paper seems to target two problems:

  1. Publication bias (only statistically significant results get published) and
  2. Bias in design calculations for new studies (taking too large expected effect sizes as benchmarks).

The good news is, both problems can be addressed in a satisfactory way.

  1. Given a plausible expected effect size $\beta^{plausible}$, an estimated effect size $\hat\beta$ (assuming it was published because it was statistically significant, while otherwise it would not have been published), an estimated standard error $s.e.(\hat\beta)$ and the distribution family (e.g. Normal or Student's $t$) of the estimator, we can backtrack the unconditional distribution of the effect size $P_{\hat\beta}(\cdot)$.
  2. Using previous findings, with the help of 1. a plausible effect size $\beta^{plausible}$ can be determined and used in study design.

To briefly answer my own two questions:

  1. It is about the publication bias, although not in a sense of data dredging but in the context of underpowered studies; there a statistically significant result is likely to belong to the, say, 5% rejections under the null (thus the null is actually true but we happened to end up far away from it by chance) rather than a rejection under the alternative (where the null is not true and the result is "genuine").
  2. I should be cautious about rejecting the null, because the statistically significant result is likely to be due to chance (even though the chance is limited to, say, 5%) rather than due to a "genuine" effect (because of low power).
$\endgroup$
  • 2
    $\begingroup$ This answer by Glen_b is also very helpful. $\endgroup$ – Richard Hardy Mar 8 '17 at 12:51
  • $\begingroup$ Idk if there's really anything non-redundant in it but I also wrote an answer to that question that might be helpful. One point: I think they don't necessarily advocate estimating the "true" distribution of the effect size using $\beta^{plausible}$ (called $D$ in the paper) but rather using it to estimate the probability of having made a Type S or Type M error based on your current test results. It's Bayesian, but IMHO sort of "Bayesian-lite" ;) because you're still using it to interpret the results of a frequentist test. $\endgroup$ – Patrick B. Mar 18 '17 at 1:31
  • $\begingroup$ @PatrickB., thank you. I will take a look a little later. (I see I had upvoted that answer of yours already before; that means I had already found it helpful.) $\endgroup$ – Richard Hardy Mar 18 '17 at 8:27
  • 1
    $\begingroup$ Richard, I have developed an R function to estimate Type "S" and Type "M" error for a more general case of effect sizes, not what Gelman shows under the normal distribution. There is as you read the paper a simple recovery process from a previously, and statistically significant finding. But the Whole process is completely based on a power analysis. In essence, for small noisy studies the SE is large and by assuming several reasonable by empirically verifiable plausible effect sizes you can obtain reasonable ... $\endgroup$ – rnorouzian Jul 31 '17 at 2:55
  • 1
    $\begingroup$ ... estimates as to what a future study should include in terms of say sample size needed to avoid getting high rates of Type "S" and high exaggeration rate (i.e., Type "M"). For the records, Gelman's Type "S" is simply that piece under the underlying effect size distribution that is on the opposite side of the underling effect divided by power. Anyway, look at the function in case it can help. $\endgroup$ – rnorouzian Jul 31 '17 at 2:59
2
$\begingroup$

There is another angle of this paper which can be helpful if you are already applying a Bayesian analysis and don't care about the statistical significance part.

Suppose $P$ is the posterior CDF of the quantity $\beta$ (effect size) you are interested in estimating. In the Bayesian situation, taking some liberty with notation and switching to talk about probability density functions, you will have a likelihood function based on some observable quantity $V$, and some pure prior of $\beta$:

$$ p(\beta | V) \sim p(V | \beta)p(\beta) $$

Here $V$ is likely to be a vector quantity, in the simplest case being a vector of multiple independent observations from which the usual product of likelihood terms arises, turning into a sum of log terms, etc. The length of that vector $V$ would be a parameterization of the sample size. In other models, say where $p(V | \beta)$ is Poisson, it might be rolled up into the Poisson parameter, which also expresses a parameterization of sample size.

Now suppose you make a hypothesis $\beta^{plausible}$ based on literature review or other means. You can use your assumed data generating process $P(V | \beta)$ with $\beta = \beta^{plausible}$ to generate simulations of $V$, which represent what data you would see if your model is well specified and $\beta^{plausible}$ is the true effect size.

Then you can do something sort of stupid: turn around and act like that sample of $V$ is the observed data, and draw a bunch of samples of $\beta$ from the overall posterior. From these samples, you can compute the statistics as mentioned in the paper.

The quantities from the linked paper, type S error and exaggeration ratio, already represent pretty much the same thing. For that effect size, given your model choices, these will tell you for a given parameter of sample size chosen for $V$, what the posterior probability of the wrong sign is and what the expected (in the posterior) ratio will be between the effect size produced by the model and the assumed plausible effect size, as you vary whatever aspect of $V$ relates to sample size.

The trickiest part is interpreting posterior "power" as the posterior probability that the estimated value of $\beta$ is at least as large as the hypothetical value $\beta^{plausible}$. This is not a measure of capacity to reject the null hypothesis, since the size of this probability would not be used as a significance measure in the frequentist sense.

I don't really know what to call it, except to say that I have had several applications in practice where it is a very helpful metric to reason about for study design. It basically offers you some way to see how much data you need to provide (assuming your data is generated perfectly from a process utilizing $\beta^{plausible}$) for a particular assumption about likelihood and prior shapes to result in some "sufficiently high" posterior probability of an effect of a certain size.

Where this has been most helpful for me in practice is in situations where the same general model needs to be repeatedly applied to different data sets, but where nuances between the data sets might justify changing the prior distribution or using a different subset of literature review to decide what's a pragmatic choice of $\beta^{plausible}$, and then getting a rough diagnostic about whether these adjustments for different data sets would result in a case where you'll need severely much more data to have non-trivial probability in the posterior concentrated in the right part of the distribution.

You have to be careful that nobody misuses this "power" metric like it is the same thing as a frequentist power calculation, which is quite hard. But all of these metrics are quite useful for prospective and retrospective design analysis even when the whole modelling procedure is Bayesian and won't refer to any statistical significance result.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.