What exactly is a prior in MAP and how do we get it?

Question

I've recently been studying MAP (Maximum a posteriori estimation). From what I've learned, the prior P(Θ) is an expected distribution of Θ, where Θ is a likelihood P(X|Θ) parameters vector.

$P(\Theta |X)=\frac{P(X|\Theta)*P(\Theta )}{P(X)}$

Therefore, if our likelihood is normal, then Θ = [μ, σ]. So the prior is kind of penalty for likelihood for choosing rare Θ (μ and σ in this case).

This makes sense, but how do we get the prior remains a mystery to me.

The general idea behind prior is to include some additional knowledge in the model.

The obvious source of this knowledge could be some research. E.g. The distribution of human height across the world. Then we could measure the height of a small group of people sampled form some island and refine the experiment result with the general knowledge of human height distribution. This sounds quite reasonable.

However the human height distribution is not the valid prior, right? We want μ and σ distribution for our prior.

1. How do we get μ and σ distribution for the prior?
2. Can we use the research form above to create a prior? How?
3. What if the research suggests a different type of distribution than our sample (e.g. research suggests Normal distribution, but out observations look like a Beta distribution and we want to model them as Beta)?

Answer 1

The way I've understood how one chooses a Bayesian prior, it can be a little hand-wavey at times. But one can be more impartial via a Jeffreys prior (discussed in a bit).

As you've said, it very often comes from one's belief on what appears to be reasonable realizations for the parameters given previous research/past experience.

Often, if you already have a distributional family you believe best describes the phenomenon in question, you could choose a conjugate prior for the model in question. Essentially, choosing a conjugate prior makes Bayesian updating "easy"/computationally simple, because the posterior distribution remains in the same distribution (just with updated parameters).

Choosing a conjugate prior may sound limiting, but a lot of probability distributions have an impressive amount of "flexibility" to them -- the Gamma distribution for variables with support $[0, \infty)$ is one example, as is the Beta distribution for a random variable with support $[0,1]$.

You may still want to use the "machinery" of Bayesian estimation but not want to "blunt" the information given by the data with your own beliefs. In that case, it would likely be easiest to choose the Jeffreys prior (see also, e.g., this thread for your distribution.

To directly answer your questions:

1. What specific prior distribution you use does ultimately end up being your choice. You may look to the reasons why previous research chose their priors or, if you want the data to "speak for itself" and not incorporate previous knowledge, use the Jeffreys prior relevant to your model.

In the example you give about heights, let's assume you have no reason to believe the population you've studied should differ significantly from some regional distribution that's modeled as $X \sim N(\mu_R, \sigma^2_R)$. If you're also modeling your population with some $N(\mu_Y, \sigma^2_Y)$, perhaps you'll choose a prior distribution $\pi(\mu_Y, \sigma^2_Y)$ such that the expectation matches the previous data, i.e. $E_{\mu_Y \sim \pi}[\mu_Y] = \mu_R$ and $E_{\sigma^2_Y \sim \pi}[\sigma^2_Y] = \sigma^2_R$

2. Previous research is a common, useful source for informing one's priors. You would likely want to use a conjugate prior for whatever distribution you're modeling the data with. To determine the prior's parameters, you would want some way of quantifying how "important" the previous research should be compared to the new dataset you've acquired.

A thought process could be "Say I've collected $n$ samples. Previous research has collected $m$ samples, but of a population that differs in some way (demographics, location, etc.). If I could somehow 'convert' this previous research into my data, how many 'effective samples' $n'$ would it be?" Then "load" your conjugate prior with the appropriate parameters to effect this weighting.

(To make the above "sample conversion" more data-driven and inform such a conversion, you could, say, perform a regression on $\text{your input variable(s)} \sim \text{previous research's input variables}$ and consider its results, e.g. its $R^2$.)

3. It would be curious for a Normal distribution (that has support in $(-\infty, \infty)$) and a Beta distribution (which has support in $[0,1]$) to both be considered feasible characterizations of the same random variable, unless the former was used for convenience and $\sim 100\%$ of the mass of the Normal distribution fell within $[0,1]$. If you feel strongly that the latter is better, I would probably load up a conjugate prior (conveniently, the Beta distribution is its own conjugate prior!) with the appropriate parameters so that the mean and variance match (or are influenced by -- see the discussion in (2)) those from previous research.