0
$\begingroup$

I am new to Bayesian parameter estimation. I have a model with a set of parameters I want to estimate from data. I know how to estimate the parameters from single data set (i.e. data of single experiment). The credibility interval I get now is totally influenced by my prior on the parameters. In order to relax this dependence, I understand that if my data is more informative, influence of prior will be minimal and the likelihood will play the major role. Fortunately, I have data sets of 10 experiments (i.e. performed the same experiment 10 times). So, my questions: 1) Is there is a way in Bayesian to combine the data of the 10 experiments together in a single regression (parameter estimation) problem so that I estimate parameters that fits the 10 data sets at once? 2) Is this consistent with Bayesian (i.e. I use the same parameters, with single Credibility interval to fit several datasets, or Bayesian is used just to estimate single parameter set from single dataset? 3) Is this method the appropriate way to overcome the role of prior, or there is other ways?

Sorry if my questions are too many and silly. I am really in bad need of your help.

Best regards

$\endgroup$
1
  • 1
    $\begingroup$ If you performed the same experiments multiple times, could you then not simply combine all data into a single dataset? $\endgroup$ Feb 2, 2017 at 8:23

2 Answers 2

3
$\begingroup$

I agree with what was said in the answer by @Tim, especially about how you need to think about the extent to which the experiments are similar or different. In effect this is (or ought to be) part of the prior information. At the two extremes you can treat the experiments as independent or as a single experiment. The former case can be called "no pooling" and the latter case can be called "complete pooling". The purpose of my answer is to point out the possibility of "partial pooling", in which the information from each experiment (each dataset) contributes to the prior for every other experiment/dataset.

This perspective is discussed in a number of texts, including Bayesian Data Analysis by Gelman and others.

The framework for this approach typically involves a hierarchical model in which the prior for the parameters (the regression coefficients in your case) involves hyperparameters along with their hyperpriors. Each dataset informs the hyperparameters, thereby providing the channel through which information flows from one dataset to another.

In effect this approach produces "shrinkage" estimates of the coefficients where the individual posterior distributions are shrunk toward a each other. The amount of shrinkage depends on the amount of similarity found during the estimation, which itself can be thought of as an exercise in "signal extraction".

Let me provide an example for explicitness. Let $Y_{1:n} = (Y_1, \ldots, Y_n)$ denote $n$ datasets of the form $Y_i = (y_{i1}, \ldots, y_{iT_i})$. In other words, there are $T_i$ observations for dataset $i$. For simplicity, let $$ p(Y_i|\theta_i) = \prod_{t=1}^{T_i} p(y_{it}|\theta_i) $$ and let $$ p(Y_{1:n}|\theta_{1:n}) = \prod_{i=1}^n p(Y_i|\theta_i) . $$ Thus far we can assumed independence in the likelihoods. Now we come to the prior for the parameters $\theta_{1:n}$.

Assume the parameters $\theta_i$ are distributed independently and identically given the the hyperparameter(s) $\psi$: $$ p(\theta_{1:n}|\psi) = \prod_{i=1}^n p(\theta_i|\psi). $$ If the hyperparameter $\psi$ is fixed at some value (say $\psi_0$), then the priors for each of the parameters $\theta_i$ are independent and no information is conveyed across datasets.

On the other hand, by providing a (non-degenerate) prior for $\psi$, we induce dependence in the prior among the $\theta_i$: $$ p(\theta_{1:n}) = \int p(\theta_{1:n}|\psi)\, p(\psi)\,d\psi . $$ It is this dependence that allows for learning across the parameters of the various datasets. You will learn (to one extent or another) as long as you are "willing to learn." This willingness is expressed in the prior. (It is not a feature of the likelihood.) From this perspective, the prior is not a burden ("I have to have a prior"), rather it is more like a blessing (I get to use a prior).

The posterior distribution for $\theta_i$ can be expressed as $$ p(\theta_i|Y_{1:n}) = \frac{p(Y_i|\theta_i)\,p(\theta_i|Y_{1:n}^{-i})}{p(Y_i|Y_{1:n}^{i})} , $$ where $Y_{1:n}^{-i} = Y_{1:n} \setminus \{Y_i\}$. In this expression, we see that the prior for $\theta_i$ is based on all the other datasets.

Of course this leaves open the question as to how exactly to structure $p(\theta_i|\psi)$ and $p(\psi)$. A approach that is both simple and natural is to take whatever prior you would have used for a single experiment/dataset and treat the parameters in that prior as your hyperparameters and put a prior on them. This approach is described in detail in Gelman's text (and other texts as well). I will refer to this a the parametric approach.

I wish to point out another more general approach that may be of use at some point, if not currently.

It is possible to interpret the problem at hand as one of latent-variable density estimation. From a Bayesian perspective, density estimation involves computing a posterior predictive distribution for an observable variable. In this case, however, we wish to compute $$ p(\theta_{n+1}|Y_{1:n}) , $$ which summarizes what is known about the parameter for the next experiment based on all of the observations for the data we have. From the perspective of density estimation, we seek a prior with sufficient flexibility to capture a wide variety of shapes including multimodality. The parametric approach mentioned above is typically too restrictive for this task.

Here is the outline of a more flexible approach. Consider the mixture model $$ p(\theta_i|\psi) = \sum_{c=1}^m w_c\,f(\theta_i|\phi_c), $$ where $\psi = (w,\phi)$, $w = (w_1, \ldots, w_m)$ are the mixture weight, $\phi = (\phi_1, \ldots,\phi_m)$ are the corresponding mixture component parameters, and $f(\theta_i|\phi_c)$ is called the kernel. The prior for the hyperparameters is given by letting $\phi_c$ be iid from some distribution $H$ and letting $w$ be drawn from the Dirichlet distribution or some "stick breaking" distribution. In fact, this class of priors includes the Dirichlet Process Mixture model.

The purpose of this digression into latent variable density estimation is not to show how to do it, but rather to indicate that such a thing is possible and show the skeletal features of the structure.

$\endgroup$
2
  • $\begingroup$ Hi mef. Thanks for reply & sorry for my late response. I have 2 issues. First, The majority of your treatment is to adjust the prior (i.e. using Hierarchical approach). My idea was to use the same prior (i.e. same parameters) but to adjust the likelihood (i.e. by including the N exp which you did in your second equation. Considering that I have nearly similar experiments, I can increase the data in the likelihood provided that they are independent experiments but use the same prior. This is a translition on my understanding about "the more informative data, the less role of prior". $\endgroup$
    – MBM
    Feb 25, 2017 at 7:10
  • $\begingroup$ The second thing is your indexing to the parameter theta. In your 1st equation you indexed it by i for T observations which means parameter for each observation. In the 2nd equation, you indexed it as i also for N_exp which is confusing to me and thus the remaining analysis as I don't understand how the parameters are treated in the single experiment and interconnect between the experiments. $\endgroup$
    – MBM
    Feb 25, 2017 at 7:13
2
$\begingroup$

What is the difference between storing data from ten experiments in ten datasets, as compared to storing it in single dataset with additional column for marking the experiment identifier? There is no difference and the choice of method for storing the data would reflect rather the practical, then statistical considerations.

As about analyzing the data, you should ask yourself if the experiments are "the same thing", or more formally, if they are exchangable. In most cases if you have multiple experiments on the same thing, then you would analyze the data in a single model, but taking into consideration also the fact that you had multiple experiments, e.g. by using additional dummy variables for marking the experiments. In such case, your model would estimate some kind of overall effect of your experiments and the individual effects for each of the experiments. If it doesn't matter what experiment does your data come from, then your model should estimate that the individual effects of experiments are close to zero, and your model would de facto simplify to model that does not use information about particular experiments. On another hand, if it makes a difference where did the data come from, then your model should be able to find such effects. In such case you should pay greater attention to finding why it matters where did the data come from (maybe the experiments were not the same, there were some problems with sticking to the experimental procedure, or with the data itself, etc.).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.