Learning prior distribution from data

Question

Suppose I have a dataset. How can I learn the prior distributions of the parameters of a model from this data? I want to learn the prior from this data in order to use them in a Bayesian model. Sorry if the question is too simple.

Answer 1

From a dataset $x_1,\ldots,x_n$, generated from a fixed distribution $f(\cdot;\theta_0)$, one can learn about the single value $\theta_0$ of the parameter that led to its generation, that is, the generation of this dataset, but close to nothing of the prior distribution of this single value, because precisely it is single: even observing $\theta_0$ exactly does not allow me to learn about its distribution $\pi$.

To construct a posterior, one need to make an informed or uninformed choice of a prior. Posteriors cannot be created ex nihilo. However, combining any prior $\pi_0$ and the data $X$ to construct a posterior $\pi_0(\theta|x)$ and then re-start with this posterior as if it was a prior is not part of the Bayesian approach because it uses the data twice: $$\pi^\text{prop}(\theta)=\frac{\pi_0(\theta) f(x|\theta) f(x|\theta)}{\int \pi_0(\theta) f^2(x|\theta)\text{d}\theta}$$ which is a proper Bayesian posterior with the data duplicated and thus multiplies by two the information brought by the data.

To drive the point about the problematic features of this proposal, repeating the "posterior-turns-prior" process over and over actually leads to a prior that is a Dirac mass at the maximum likelihood estimator associated with the data and the model. (This may also prove an alternative scheme to obtain the maximum likelihood estimator. That we called State-Augmentation for Marginal Estimation (SAME), lated copied under the new name of data cloning.)

This illustrates most forcibly how the substitution creates a bias or an over-fit in the analysis and there are several drawbacks well documented in the literature, even though some speak of "empirical Bayes" (Robbins, 1953; Carlin and Louis, 1996) "data-dependent priors" (Wasserman, 2000) or "integrated likelihoods" (Aitkin, 2009). To quote from a recent thesis by W. Darnieder (Ohio State U):

Curiously, the data-dependent strategy attempts to circumvent the often difficult task of establishing prior belief while still masquerading as a Bayesian technique. In so doing, the data-dependent Bayesian will incorporate aspects of the observed data into his assessment of prior information before experimentation (or at least before analysis), and in the formal posterior updating in accordance with Bayes Theorem. This double use of the data within the Bayesian paradigm is highly objectionable because it can easily be abused to produce virtually any outcome the researcher de-sires (Gelman et al., 2003). This is true even when the researcher is not performing a deliberately deceptive analysis, but when the researcher may unwittingly be using a data-dependent prior so strong that it completely drives posterior inference (...)

Similarly, those critical of data-dependent techniques can originate their arguments with the Likelihood Principle which states that all experimental information is contained in the data likelihood. Generally speaking, empirical Bayes techniques strongly violate this principle. Many objective Bayesian methods also violate the Likelihood Principle, as we have seen in the examples of Jeffreys, Zellner, and Wasserman,but the scope of offense is frequently venial by comparison, e.g. when an objective prior depends only on the sample size. (...) And yet the explicitly data-dependent approach still seems worse because of its brazen disregard of such a principled approach while still invoking the name Bayes.

The above criticisms notwithstanding, the most odious feature of the typical use of data-based priors is how the elegant simplicity of Bayes’ Theorem has been perverted. The statement $p(θ|x)∝f(x|θ)π(θ)$ is true and the usual rules of probability remain in force, while $p(θ|x)∝f(x|θ)π(θ|T(x))$ is patently false. Deely and Lindley (1981) humorously commented that, “It is this use of Bayes formula that presumably accounts for the reference to Bayes in the term, empirical Bayes.” Worse, the data-dependent formulation completely abandons the rules of probability.

A side remark is that Bayesian posteriors can be sequentially updated in the sense that the posterior $\pi(\theta|x_1,\ldots,x_n)$ is also the posterior associated with the prior $\pi(\theta|x_1,\ldots,x_{n-1})$ and the additional observation $x_n$, with $\pi(\theta|x_1,\ldots,x_{n-1})$ itself being the posterior associated with the prior $\pi(\theta|x_1,\ldots,x_{n-2})$ and the additional observation $x_{n-1}$, &tc., &tc.