Understanding Gelman & Carlin "Beyond Power Calculations: ..." (2014) I am reading Gelman & Carlin "Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors" (2014). I am trying to understand the main idea, the main takeway, but I am confused. Could anyone help distill me the essence?
The paper goes something like this (if I understood it correctly).


*

*Statistical studies in psychology are often plagued by small samples. 

*Conditional on a statistically significant result in a given study,
(1) the true effect size is likely to be severely overestimated and
(2) the sign of the effect may be opposite with high probability -- unless the sample size is large enough.

*The above is shown using a prior guess of the effect size in population, and that effect is typically taken to be small.


My first problem is, why condition on the statistically significant result? Is it to reflect the publication bias? But that does not seem to be the case. So why, then?
My second problem is, if I do a study myself, should I treat my results differently than I am used to (I do frequentist statistics, not very familiar with Bayesian)? E.g. I would take a data sample, estimate a model and record a point estimate for some effect of interest and a confidence bound around it. Should I now mistrust my result? Or should I mistrust it if it is statistically significant? How does any given prior change that?
What is the main takeaway (1) for a "producer" of statistical research and (2) for a reader of applied statistical papers?
References:


*

*Gelman, Andrew, and John Carlin. "Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors." Perspectives on Psychological Science 9.6 (2014): 641-651.


P.S. I think the new element for me here is the inclusion of prior information, which I am not sure how to treat (coming from the frequentist paradigm).
 A: I re-read the paper and this time it seems much clearer. Now also the helpful comments by @Glen_b and @amoeba make lots of sense.
The whole discussion is based on a starting point that a statistically significant result has been obtained. Conditional on that, we have the estimated effect size distributed differently than it would be absent the conditioning: 
$$
P_{\hat\beta}(\cdot|\hat\beta \text{ is statistically significant})\neq P_{\hat\beta}(\cdot).
$$
The paper seems to target two problems:


*

*Publication bias (only statistically significant results get published) and 

*Bias in design calculations for new studies (taking too large expected effect sizes as benchmarks). 


The good news is, both problems can be addressed in a satisfactory way. 


*

*Given a plausible expected effect size $\beta^{plausible}$, an estimated effect size $\hat\beta$ (assuming it was published because it was statistically significant, while otherwise it would not have been published), an estimated standard error $s.e.(\hat\beta)$ and the distribution family (e.g. Normal or Student's $t$) of the estimator, we can backtrack the unconditional distribution of the effect size $P_{\hat\beta}(\cdot)$. 

*Using previous findings, with the help of 1. a plausible effect size $\beta^{plausible}$ can be determined and used in study design.


To briefly answer my own two questions:


*

*It is about the publication bias, although not in a sense of data dredging but in the context of underpowered studies; there a statistically significant result is likely to belong to the, say, 5% rejections under the null (thus the null is actually true but we happened to end up far away from it by chance) rather than a rejection under the alternative (where the null is not true and the result is "genuine").

*I should be cautious about rejecting the null, because the statistically significant result is likely to be due to chance (even though the chance is limited to, say, 5%) rather than due to a "genuine" effect (because of low power).

A: There is another angle of this paper which can be helpful if you are already applying a Bayesian analysis and don't care about the statistical significance part.
Suppose $P$ is the posterior CDF of the quantity $\beta$ (effect size) you are interested in estimating. In the Bayesian situation, taking some liberty with notation and switching to talk about probability density functions, you will have a likelihood function based on some observable quantity $V$, and some pure prior of $\beta$:
$$ p(\beta | V) \sim p(V | \beta)p(\beta) $$
Here $V$ is likely to be a vector quantity, in the simplest case being a vector of multiple independent observations from which the usual product of likelihood terms arises, turning into a sum of log terms, etc. The length of that vector $V$ would be a parameterization of the sample size. In other models, say where $p(V | \beta)$ is Poisson, it might be rolled up into the Poisson parameter, which also expresses a parameterization of sample size.
Now suppose you make a hypothesis $\beta^{plausible}$ based on literature review or other means. You can use your assumed data generating process $P(V | \beta)$ with $\beta = \beta^{plausible}$ to generate simulations of $V$, which represent what data you would see if your model is well specified and $\beta^{plausible}$ is the true effect size. 
Then you can do something sort of stupid: turn around and act like that sample of $V$ is the observed data, and draw a bunch of samples of $\beta$ from the overall posterior. From these samples, you can compute the statistics as mentioned in the paper.
The quantities from the linked paper, type S error and exaggeration ratio, already represent pretty much the same thing. For that effect size, given your model choices, these will tell you for a given parameter of sample size chosen for $V$, what the posterior probability of the wrong sign is and what the expected (in the posterior) ratio will be between the effect size produced by the model and the assumed plausible effect size, as you vary whatever aspect of $V$ relates to sample size.
The trickiest part is interpreting posterior "power" as the posterior probability that the estimated value of $\beta$ is at least as large as the hypothetical value $\beta^{plausible}$. This is not a measure of capacity to reject the null hypothesis, since the size of this probability would not be used as a significance measure in the frequentist sense.
I don't really know what to call it, except to say that I have had several applications in practice where it is a very helpful metric to reason about for study design. It basically offers you some way to see how much data you need to provide (assuming your data is generated perfectly from a process utilizing $\beta^{plausible}$) for a particular assumption about likelihood and prior shapes to result in some "sufficiently high" posterior probability of an effect of a certain size.
Where this has been most helpful for me in practice is in situations where the same general model needs to be repeatedly applied to different data sets, but where nuances between the data sets might justify changing the prior distribution or using a different subset of literature review to decide what's a pragmatic choice of $\beta^{plausible}$, and then getting a rough diagnostic about whether these adjustments for different data sets would result in a case where you'll need severely much more data to have non-trivial probability in the posterior concentrated in the right part of the distribution.
You have to be careful that nobody misuses this "power" metric like it is the same thing as a frequentist power calculation, which is quite hard. But all of these metrics are quite useful for prospective and retrospective design analysis even when the whole modelling procedure is Bayesian and won't refer to any statistical significance result.
