Bayesian prior via cross validation

Question

I have a particular problem where I am using Bayesian techniques to estimate parameters of a distribution of a random variable.

I would like to use an external source of data to determine an appropriate prior distribution for the analysis (which will then be updated with internal data). Let's denote this external data as

$$\boldsymbol{X}=\{X_{1},X_{2},\ldots,X_{n}\}$$

Now, ideally I would like to use this data to determine the prior distribution of an underlying parameter vector

$$\boldsymbol{\Theta}=\{\theta_{1},\theta_{2}\}$$

My current approach is:

• For $m=\{1,\ldots,M\}$ iterations:

  1. Take a random subset of $\boldsymbol{X}$, denote this as $\boldsymbol{X}_{m}$.
  2. Estimate the parameters using $\boldsymbol{X}_{m}$, giving $\hat{\boldsymbol{\Theta}}_{m}$.

The above process provides $M$ estimates of $\boldsymbol{\Theta}$ and provides a prior distribution for our analysis.

I feel this a reasonable approach as it should be quite robust as we have randomly sampled from our original data each time we estimate. Obviously, the analysis will depend on the size of the subsample. I view this as some sort of cross-validation implicit in the estimation.

Is the above a reasonable approach to determining a prior distribution from an external source of data?

Additional Essentially, I want the external data to construct prior distributions for each of the parameters. The external data contains information that the internal data is lacking and therefore would like the priors to be a starting point for the Bayesian analysis which the internal estimates will be based on.

Just for clarification, the model had been set up as follows:

where the iterative procedure is given below:

Answer 1

Yes, it is an excellent method. Usually, priors are built from less. That is a very defensible solution to constructing a prior density.

EDIT See Bayesian Data Analysis, Third Edition by Andrew Gelman, John B. Carlin, Hal S. Stern, David B. Dunson, Aki Vehtari, Donald B. Rubin

see the segments on the use of historical data and informative priors in the index

See also Introduction to Bayesian Statistics by William M Bolstad and James M Curran. See the chapter on robust Bayesian methods.

There is an article by Jaynes at http://bayes.wustl.edu/etj/articles/highly.informative.priors.pdf on highly informative priors.

There is also this article at Hamra, Ghassan et al. “Integrating Informative Priors from Experimental Research with Bayesian Methods: An Example from Radiation Epidemiology.” Epidemiology (Cambridge, Mass.) 24.1 (2013): 90–95. PMC. Web. 5 Apr. 2018.

EDIT How interesting, I am working on a similar problem assuming LGD is loss given default.

So, since I do not know your methodology, but I presume it is some form of regression, I would recommend looking at the Bolstad method of mixing prior results from the literature with a flat prior. These leaves the center of location intact but spreads out the uncertainty.

To provide an example, let us imagine you know that $\hat{\beta}_x=1.23$ in the literature with a variance of $.000001$ so it is estimated to a precision less than your least significant digit. You could construct a prior distribution around the literature's center of location but with a high variance. How high is high would, of course, depend upon the scaling of your literature, but you want it to be high enough that you have captured all reasonable estimates of the parameter in the dense region of your prior.

This is the difficulty of Bayesian methods, they depend on judgment to some extent. How the joint distribution of parameters works will also be a judgment call. While you could take the naive Bayes approach of no covariance if there is a published covariance matrix then it should be the starting point of your search.

What you are wanting your Bayesian method to do is guide your estimator into the region that is most likely the true region. You should add variance to your estimators to account for the cultural and legal differences surrounding losses.

I would love to give you a clear answer that every editor in the journals would take, but the most you can do is hunt the statistical literature for "informative prior" and "highly informative prior." There would also be content on robust methods and mixture priors.

Gelman's section on the use of historical data would also be of use. He anticipates a body of literature from which to construct the prior. This is problematic since no one is actually performing the same exact research in question.

One approach that could be taken is to grab Cox's axiomatic approach where probability is grounded in logic. Hence, you need to construct a logical framework for the construction of the prior. Hence your real job is disclosure and reasonableness and not a precise algebraic formulation.

Answer 2

If you have $n$ samples, then you can use Bayesian updating, to update your priors sequentially, by starting with a prior $\pi(\Theta)$, then you can use Bayes theorem to estimate the posterior

$$ \pi_1(\Theta) = \pi (\Theta | X_1) \propto p(X_1 | \Theta)\; \pi(\Theta) $$

to update your knowledge given $X_2$ sample, you take

$$ \pi_2(\Theta) = \pi (\Theta | X_1) \propto p(X_2 | X_1, \Theta)\; \pi_1(\Theta) $$

etc., using the general

$$ \pi_{n+1}(\Theta) = \pi (\Theta | X_{n+1}) \propto p(X_{n+1} | X_1, \dots, X_n, \Theta)\; \pi_n(\Theta) $$

what would be equivalent to updating all at once

$$ \pi(\Theta|X_1,\dots,X_n) \propto p(X_1,\dots,X_n|\Theta) \; \pi(\Theta) $$

To give an example, consider a beta-binomial model, where the posterior distribution is

$$ \alpha,\beta|x \sim \mathcal{B}(\alpha + x, \,\beta + n-x) $$

so if you had two samples of sizes $n_1,n_2$ and observed $x_1,x_2$ successes, then first you'd update the prior parameters $\alpha,\beta$ to $\alpha+x_1$ and $\beta+n_1-x_1$, and then, using second sample, to $(\alpha+x_1)+x_2$ and $(\beta+n_1-x_1)+n_2-x_2$, what would be equivalent to updating all-at-once: $\alpha + (x_1 + x_2)$ and $\beta + (n_1 + n_2) - (x_1 + x_2)$.

This directly translates to your case, where you have the "external" $X_1,\dots,X_n$ samples and want to use them to generate posterior for the "internal" sample $X_{n+1}$. Basically, this is the general idea behind Bayesian approach, where you can use your initial, or previous, knowledge and include it into your model as a prior, that is updated given new data.

What follows, is that

• using posterior obtained using $X_1,\dots,X_n$ to update given $X_{n+1}$ is a perfectly valid way to go,
• if for some reasons you need to proceed sequentially (e.g. the data comes sequentially), then this is a valid way to go,
• it doesn't make much sense to obtain the $n$ independent "prior" estimates of $\Theta$ given $n$ samples, since you can proceed all-at-once,
• in fact, you shouldn't make $n$ independent estimates using the same prior in each case, since if then you somehow aggregated the results, then in the final result you'd include your prior $n$ times,
• you shouldn't use the same data to "estimate" the prior (so $X_{n+1}$ really needs to be new data), since then the same information would be used twice and you would end up with the result that is overconfident (while the point estimates wouldn't change, the posterior distributions and interval estimates would be too narrow).

On another hand, if what you are asking is if using frequentist approach on the external data, to estimate the parameters for the priors, is a valid way to go, then the answer is still: yes, we often use external data, or previous results to create informative priors. However a Bayesian would still argue, that there is no reason for using frequentist approach in here. You can go all the way using Bayesian approach: starting with some prior, updating it using the external data, then using the posterior as a prior for a model using the "internal" data.

Answer 3

This thread is somewhat old, but this is a problem that I have been giving some thought to and wanted to suggest an approach.

Let's say you have external data $X_1, \ldots, X_n$ and internal data $X_{n+1}, \ldots, X_m$, a likelihood function $p(X_1, \ldots, X_n \mid \theta)$ and a prior distribution $\pi(\theta \mid h)$. Your goal is to use the external data to estimate $h$, then update your prior using the internal data.

You could do this using an empirical Bayes type approach. That is, you first calculate the likelihood of the hyperparameters $h$, given the external data. Then, you estimate $h$ using maximum likelihood estimation. $$p(X_1, \ldots, X_n \mid h) = \int p(X_1, \ldots, X_n \mid \theta)\pi(\theta \mid h) d\theta$$ $$\hat{h} = \max_h p(X_1, \ldots, X_n \mid h)$$

You then proceed in the usual Bayesian way, updating the prior based on the internal data.

$$\pi(\theta \mid \hat{h}, X_{n+1}, \ldots, X_m) \propto p(X_{n+1}, \ldots, X_m \mid \theta) p(\theta \mid \hat{h})$$