7
$\begingroup$

I'm not clear on how to perform sensitivity analysis on the priors. Many sites have different answers. One site indicates to perform three non-informative, weakly informative and known priors. Another suggest running the model with different priors.

Here are my questions:

I want to estimate this parameter A, in which A = mean1 + mean2 . So, the priors for mean1 and mean2 are formulated (in coding) as the following :

mean1 ~ N(u1, tau1)
A ~ N(mean1, tau2) 

where

tau1 ~ gamma(alpha1,beta1)
tau2 ~ gamma(alpha2,beta2)

With the above formulation, how do I perform the sensitivity analysis?

  1. Should I define a range of u1, tau1, and tau2 ? If so, what is the systematic procedures? It seems to me there are various combinations: varying u1 and fixing tau1 and tau2; or fix u1 and tau2, varying tau1; or fix u1 and tau1, varying tau2. How do we know what are the correct priors to use?

  2. How do I incorporate weakly informative priors to the above formulation?

  3. What if the sensitivity test returns different estimated mean1 and A by varying the priors? How do we make conclusion?

Your advice will be appreciated and help me get started. Thank you!

EDIT: I corrected an error. You can calculate the estimated mean2 from estimated parameters A and estimated mean1.

$\endgroup$
  • $\begingroup$ Excellent questions, I asked myself similar questions during a long time. Finally the answer I found was the abandonment of subjective priors. These questions are endless. More seriously, perhaps there are answers more positive than mine but I think that should depend on the professional context. $\endgroup$ – Stéphane Laurent Mar 25 '14 at 22:49
3
$\begingroup$

Try as I might, I'm unable to find an open source that fully explains that process, so for a fuller treatment than the below, I'm left to direct your to Bayesian Data Analysis. In particular, Ch. 6 should prove very helpful.

In broadest strokes, following the analysis one should ponder whether model inferences make realistic sense. For instance, consider this paper detailing prior considerations in a pharmacological study. In brief, liver size was a variable relevant to the model, about which much is known in medical literature:

For example, parameter 8 represents the mass of the liver as a fraction of lean body mass; from previous medical studies, the liver is known to be about 3.3% of lean body mass for young adult males, with little variation.

A non-informative prior would have suffered a few pitfalls:

If noninformative prior distributions were assigned to all the individual parameters, then the model would fit the data very closely but with scientifically unreasonable parameters – for example, a person with a liver weighing 10 k. This sort of difficulty is what motivates a researcher to specify a prior distribution using external information.

This is a simple test of realism: If your model suggests a human liver the size of a Thanksgiving turkey, there's a flaw of some sort.

While not well-applied to your problem, this example makes clear how much these considerations depend on context. To examine whether the posterior is overly dependent on the prior, one can consider multiple priors and see whether the posterior changes, in a practical sense, as different priors will necessarily always yield different posteriors. For example, say you're polling a constituency about support for a proposed law. In particular, no one has ever gathered data about this population's support for this law: Your data exist in a knowledge vacuum.

You construct a model in which each person you randomly poll is a Bernoulli random variable with parameter $\theta \le 1$. You select the beta distribution as a convenience prior for $\theta$, because it's conjugate to the Bernoulli. You've polled 98 randomly selected persons, 45 of which support the measure. But you wonder whether the Jeffrey's prior, $B(\frac{1}{2}, \frac{1}{2})$ or a uniform density over $[0,1]$, given by $B(1,1)$. Leaving the formal math aside, the respective posteriors are $B(45.5, 53.5)$ and $B(46, 54)$, and it's difficult to see this difference could be of practical consequence:

Two beta distributions B(45.5, 53.5) and B(46, 54) in green and blue, respectively

(Note that the blue and green lines are nearly indistinguishable.) Now, you can imagine that if one had previous, extensive polling data on this proposal, then perhaps this data could signal an change in opinion, a status quo result, or random noise. In that case, one would be wise to compare these inferences with ones derived from more informed priors.

Now, to your specific questions above:

  1. Parameters of interest must always have a specified distribution and prior, lest one couldn't make inferences on them at all. To the question of correct priors, that again depends on the state of knowledge of the problem, and perhaps what priors your skeptical audience will find agreeable. If knowledge is scant, you may wish to consider non-informative priors. But any choice of prior, or the choice to fix a parameter, must be justified either from existing knowledge or uncertainty.
  2. Simply derive posteriors from any priors you would like to consider, then compare them to examine whether the difference is of consequence to the problem at hand.
  3. That's the tough one, as it depends entirely on context. If any prior produces a turkey-liver type inference, it's probably safe to dismiss. But for subtler distinctions, I'm aware of no substitute for subject matter expertise, careful analysis, and more data. Bayes factors are often used in model comparison, but typically when comparing models of distinct forms. (I've honestly never considered Bayes factors to compare priors, but my sense is that a Bayes factor analysis would favor the vaguer prior, i.e. the one that gave more weight to the data.)
$\endgroup$
  • $\begingroup$ Thank you for the detail explanation and the links. I'm still at the learning stage. Few months ago, I tried to understand Bayes factor but no avail in how to put the theory into pymc code(practical). So, the idea of using Bayes factor was gave up. $\endgroup$ – user3460430 Apr 15 '14 at 12:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.