5
$\begingroup$

US FDA authorizes the use of Bayesian statistics with informative priors (in certain contexts):

http://www.outsourcing-pharma.com/Clinical-Development/US-FDA-says-Bayesian-analysis-could-cut-device-trial-costs

http://www.stat.rutgers.edu/iob/bioconf09/slides/Campbell.pdf

FDA Bayesian guidance

In the slide "Importance of Simulation" of Campbell's slides, it is written: "So simulate to show that Type 1 error (or some analog of it) is well-controlled".

However Type 1 error is not well-controlled with an informative prior. For instance consider a simple binomial model $x \sim \text{Bin}(n,\theta)$ with a Beta prior $B(a,b)$ on the unknown proportion parameter $\theta$, and consider a credibility interval $I(x)$ for some given credibility level $100(1-\alpha)\%$. Then the frequentist coverage function $\theta \mapsto \Pr(I(x) \ni \theta \mid \theta)$ is close to $100(1-\alpha)\%$ for the Jeffreys prior $B(a=\frac12,b=\frac12)$, but when the sample size is not large and $(a,b)$ is far from $(\frac12,\frac12)$ then the frequentist coverage is far from $100(1-\alpha)\%$ (possibly except for some very particular values of $\theta$, but anyway the coverage is not "controlled"). The Type 1 error of Bayes factor tests is not controlled too.

So according to which point of view could one achieve good frequentist properties with Bayesian inference under an informative prior ?

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ Remark: I think it would be nice to have a tag for "regulatory statistics" (for questions related to FDA requirements/recommendations, for instance) $\endgroup$
    – Stéphane Laurent
    Mar 7, 2013 at 22:01
  • 1
    $\begingroup$ I agree with the suggestion for the new tag. Or maybe just make it "FDA". $\endgroup$
    – Harvey Motulsky
    Mar 8, 2013 at 0:33
  • 1
    $\begingroup$ @HarveyMotulsky There are others regulatory administrations/agencies (such as EMA) hence I'd prefer a "regulatory statistics" tag. Maybe this would be a good subject for meta.stats.stackexchange but my english is not sufficiently developed to open the discussion. $\endgroup$
    – Stéphane Laurent
    Mar 8, 2013 at 8:38

1 Answer 1

Reset to default
4
$\begingroup$

I have spoken with Dr Campbell and many others about this. The whole notion of advocating for a demonstration of type I error control is ill-defined. In the Bayesian context there is officially no such thing as type I error, and it makes us less rational and more confused to even consider it. You could say that type I error is in direct conflict with a prior that does not have a discontinuity at the null. This is a political, educational, and inertial battle.

To make matters worse, to compute type I error you have to design a sample space, e.g., you have to specify the times of intended looks at the data. This is an anathema to Bayesians.

Improve this answer
$\endgroup$
9
  • $\begingroup$ Hello Frank. So does that mean that FDA will never accept an informative prior ? (in Campbell's slides I have seen a link to an informative hierarchical Bayesian model accepted by FDA but I don't understand the context). By the way I give a link to a paper in which I show on an example how frequentist properties are broken when including information about the nuisance parameter only orbi.ulg.ac.be/handle/2268/77820 $\endgroup$
    – Stéphane Laurent
    Mar 8, 2013 at 6:30
  • $\begingroup$ Sorry my above comment is not clear (I was just waking up). We could say that the Type 1 error is "controlled" when the Bayesian inference has good frequentist properties, but as I said in my OP this doesn't occur with an informative prior. I am confused by Frank's answer: why did Dr Campbell write this slide if he claims himself that this is not sensible ? And since the frequentist properties fail, what criteria the FDA would be based on to accept an informative Bayesian analysis ? $\endgroup$
    – Stéphane Laurent
    Mar 8, 2013 at 8:35
  • $\begingroup$ No it doesn't mean that FDA will not accept an informative prior. This approach is often welcomed in device trials, given that the previous studies are relevant to the current question. It's just that the concept of type I error is confusing and the FDA guidance, while specifying they want you to show the type I error, doesn't show you how to do it (and it could be said to be impossible if you use an informative prior). $\endgroup$
    – Frank Harrell
    Mar 8, 2013 at 13:19
  • 2
    $\begingroup$ One way to do it is to find a seemingly applicable study then discount it by doubling the variance of its point estimate or by making the prior a mixture of a skeptical prior and the previous results. The mixing proportion is an "applicability quotient." $\endgroup$
    – Frank Harrell
    Mar 8, 2013 at 15:17
  • 2
    $\begingroup$ Sorry I don't have one. The gbayes function in the R Hmisc package has some relevant examples. $\endgroup$
    – Frank Harrell
    May 26, 2013 at 21:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.