# On FDA guidance about Bayesian practice

US FDA authorizes the use of Bayesian statistics with informative priors (in certain contexts):

http://www.outsourcing-pharma.com/Clinical-Development/US-FDA-says-Bayesian-analysis-could-cut-device-trial-costs

http://www.stat.rutgers.edu/iob/bioconf09/slides/Campbell.pdf

FDA Bayesian guidance

In the slide "Importance of Simulation" of Campbell's slides, it is written: "So simulate to show that Type 1 error (or some analog of it) is well-controlled".

However Type 1 error is not well-controlled with an informative prior. For instance consider a simple binomial model $x \sim \text{Bin}(n,\theta)$ with a Beta prior $B(a,b)$ on the unknown proportion parameter $\theta$, and consider a credibility interval $I(x)$ for some given credibility level $100(1-\alpha)\%$. Then the frequentist coverage function $\theta \mapsto \Pr(I(x) \ni \theta \mid \theta)$ is close to $100(1-\alpha)\%$ for the Jeffreys prior $B(a=\frac12,b=\frac12)$, but when the sample size is not large and $(a,b)$ is far from $(\frac12,\frac12)$ then the frequentist coverage is far from $100(1-\alpha)\%$ (possibly except for some very particular values of $\theta$, but anyway the coverage is not "controlled"). The Type 1 error of Bayes factor tests is not controlled too.

So according to which point of view could one achieve good frequentist properties with Bayesian inference under an informative prior ?

• Remark: I think it would be nice to have a tag for "regulatory statistics" (for questions related to FDA requirements/recommendations, for instance) – Stéphane Laurent Mar 7 '13 at 22:01
• I agree with the suggestion for the new tag. Or maybe just make it "FDA". – Harvey Motulsky Mar 8 '13 at 0:33
• @HarveyMotulsky There are others regulatory administrations/agencies (such as EMA) hence I'd prefer a "regulatory statistics" tag. Maybe this would be a good subject for meta.stats.stackexchange but my english is not sufficiently developed to open the discussion. – Stéphane Laurent Mar 8 '13 at 8:38

• Sorry I don't have one. The gbayes function in the R Hmisc package has some relevant examples. – Frank Harrell May 26 '13 at 21:14