3
$\begingroup$

Let's say I have a distribution for a random variable S:

s | P(S=s)
--+-------
0 | .28
1 | .72

That's a prior, right? It represents our belief about the likelihood of an event happening absent other information. It is fundamentally different from something like P(S=s|R=r), which represents our belief about S given exactly the information R.

Alternatively, I could be given a joint distribution for S and R and compute the marginal probabilities:

s | r | P(S=s, R=r)
--+---+------------
0 | 0 | 0.2
0 | 1 | 0.08
1 | 0 | 0.7
1 | 1 | 0.02

So the marginals are:

s | P(S=s)    r | P(R=r)
--+-------    --+-------
0 | .28       0 | 0.9
1 | .72       1 | 0.1

My question is: how is this marginal distribution for S any different than the prior for S? Is it only a matter of interpretation?

$\endgroup$
  • 1
    $\begingroup$ As Tim just said, probability theory does not distinguish between prior and marginal. Statistics does because parameter comes before or prior to the data. $\endgroup$ – Xi'an Dec 2 '16 at 15:36
  • $\begingroup$ That makes sense, and answers my question, "Is it only a matter of interpretation?" Mathematically there is no distinction. It is a matter of Bayesian interpretation, correct? $\endgroup$ – gwg Dec 2 '16 at 15:42
8
$\begingroup$

$P(S=s)$ and $P(R=r)$ both are marginal probabilities from the following table

$$ \begin{array}{c|cc|c} & R=0 & R=1 \\ \hline S=0 & 0.20 & 0.08 & 0.28 \\ S=1 & 0.70 & 0.02 & 0.72 \\ \hline & 0.90 & 0.10 & \end{array} $$

Given such table, you can calculate conditional probabilities $P(S \mid R)$, or $P(R \mid S)$ by applying Bayes theorem, e.g.

$$ P(S \mid R) = \frac{P(R \mid S) \, P(S)}{P(R)} = \frac{P(R \cap S)}{P(R)} $$

the same way you could calculate $P(R \mid S)$. Notice that to apply it you need to know either conditional, or joint probabilities. This is a basic application of Bayes theorem and it has many nice applications (see e.g. here).

Now important thing to notice: applying Bayes theorem is not the same as using Bayesian statistics. $P(S)$ in your example is not more prior, then $P(R)$. Moreover, to calculate the "posterior" probability you need to know the joint or conditional probabilities. If you are thinking of some simple example like "there is 0.7 probability that Jack has stolen an orange from the shop", you cannot apply Bayes theorem to such problem by assuming that in your opinion the probability is, for example, 0.3, unless you also know the joint probabilities (probability that he is guilty when you assume he is etc.), or conditional probabilities (probability that you assume that he is guilty given the fact that he is guilty). This is not the way how we use priors in statistics.

When applying Bayes theorem in statistics we have some data $X$ that can be described using probability density function $f_\theta$, but we do not know the value of it's parameter $\theta$. To estimate $\theta$ we can use many different statistical approaches, for example, maximum likelihood estimation by maximizing the likelihood function

$$ \DeclareMathOperator*{\argmax}{arg\,max} \argmax_{\theta} f_\theta( X ) $$

Other approach to the problem is to include some prior information into the process of estimating the parameter and use a Bayesian approach. This is done by using Bayes theorem, but in a different way. First, we assume some probability distribution for $\theta$, let's call it $g$, and then assume a priori that the unknown parameter follows this distribution. We use Bayes theorem to combine the two sources of information: our a priori assumptions about $\theta$, that is our prior $g$; and the information that is contained in the data, that is likelihood function $f_\theta(X)$, so to obtain posterior estimate $g(\theta | X)$:

$$ g(\theta | X) \propto f_\theta(X) \, g(\theta) $$

If this still sounds complicated, you can start by going through multiple other questions tagged for lots of examples. There is also many good introductory books to start with, e.g. Bayesian Data Analysis by Andrew Gelman et al, or Doing Bayesian Data Analysis, Second Edition: A Tutorial with R, JAGS, and Stan by John K. Kruschke.

$\endgroup$
  • 1
    $\begingroup$ +1 really thank you for detailed answer for such a basic question! $\endgroup$ – hxd1011 Dec 2 '16 at 16:12
2
$\begingroup$

If you think of a table with columns as possible parameter values and with rows as possible data values (see image below), then the lower marginal distribution is the prior distribution on the parameter. The observed data indicate which row of the table is the row we actually live in, so to speak, hence we conditionalize on that row to find p(theta|D), which is the posterior distribution. In other words, Bayes rule gets us from the marginal (i.e., the prior), to the row-conditional (i.e., the posterior).

enter image description here

I realize that's a pretty terse description, but I hope it helps some. For full details, please see Chapter 5 of DBDA2E.

$\endgroup$
  • $\begingroup$ Please register & merge your accounts 1 2 (you can find information on how to do this in the My Account section of our help center). $\endgroup$ – gung Dec 10 '16 at 20:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.