3
$\begingroup$

Learning Bayesian decision theory (specifically in Machine Learning) recently, couldn't figure out what do the posterior possibility $P(c|x)$ and the prior possibility $P(x|c)$ mean exactly.

Anybody knows what $x$ and $c$ represent exactly in the possibility formula? (all I know is c stand for class and x stand for samples. But I suppose they have some different meaning in the formula, perhaps represent some events?)

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ Possibility and probability are different things, while you seem to be using them as synonyms stats.stackexchange.com/q/445832/35989 $\endgroup$
    – Tim ♦
    Mar 22 at 6:45
  • $\begingroup$ @Tim Thanks for editting my question! My comprehension from your reference is that probability is a quantified representation of possibility. Do I get the point? $\endgroup$
    – Bog
    Mar 22 at 9:55

3 Answers 3

Reset to default
3
$\begingroup$

$x$ and $c$ in Bayes theorem are random variables. Any random variables. Bayes's theorem is about being able to flip sides of the conditional distribution from $P(x|c)$ to $P(c|x)$ or the other way around. They could be events, e.g. “probability that it rains ($x$) given that it’s cloudy ($c$), $P(x|c)$”, in naive Bayes algorithm $x$ is a feature of the model and $c$ is the predicted class, in a classical Bayesian model $x$ would be your data and $c$ the parameter of the model.

Improve this answer
$\endgroup$
2
  • $\begingroup$ The question is more about machine learning. And after reading some materials, I think c stands for given an arbitrary sample (sample here means something like training examples instead of the ''sample'' in statistics) , the sample belongs to a class c. And x means given an arbitrary sample, the sample is identical to x. What do you think? $\endgroup$
    – Bog
    Mar 22 at 10:16
  • 1
    $\begingroup$ @Bog I'm not sure what do you mean. $\endgroup$
    – Tim ♦
    Mar 22 at 10:43
2
$\begingroup$

Let us give an example that makes this as simple as possible.

Suppose your samples are all taken from a Bernoulli distribution, i.e. these are ``binary samples'', these are just $1$'s and $0$'s. Let those samples by denoted by: $x_1,x_2,...,x_n$. Here each $x_k$ is equal to either $1$ or $0$. Let $\mathbf{x}$ denote the entire vector of all those samples, so $\mathbf{x} = (x_1,x_2,...,x_n)$.

Now, since these samples are coming from a Bernoulli distribution it means there is an unknown parameter $c$ which represents the ``success rate''. The number/parameter $c$ is unknown, and it is supposed to represent how often the samples display $1$. Therefore, if $c=.9$ then we expect to see a lot of $1$'s in the vector $\mathbf{x}$ and if $c=.1$ then we expect to see mostly zeros instead.

Let us say, for example, that our sample vector $\mathbf{x}$ consists of $70$ observations of "$1$" and $30$ observations of "$0$". It is reasonable to guess that $c = \frac{70}{100} = .7$. However, because of random flucations in the data it could happen that the true value of $c=.65$ and the data was just more lucky and generated a bit more $1$'s.

The "posterior distribution", which you denote as $P(c|\mathbf{x})$ is supposed to quantify your uncertainty about the value of the $c$ parameter. In the example we are using, where $\mathbf{x}$ has 70 successes and 30 failures, it can be shown (perhaps, you can show this yourself!), that the posterior distribution looks like this, enter image description here

From this picture you can see that the most reasonable choice for $c$ is $0.7$, but it could also be $0.8$ but much less likely, however when we reach $0.9$ it becomes very unreasonable. Instead of saying "more reasonable", or "less reasonable", ect, we make the language precise so there is no confusion about what we mean, and the posterior distribution is what quantities your uncertainty and likelihood of the unknown parameter $c$.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Assuming you have $c$ and $x$ the correct way round, then $p(c)$ would be the prior density for $c$, and $\mathbb P(\mathbf{x} \mid c)$ would be the Bernoulli or binomial probability for $\mathbf{x}$ given $c$ which is proportional to the likelihood, and combined would give $p(c \mid \mathbf{x}) =\dfrac{\mathbb P(\mathbf{x} \mid c) \,p(c)}{\int_c \mathbb P(\mathbf{x} \mid c) \,p(c)\, dc }$ as the posterior density for $c$ given $\mathbf{x}$ $\endgroup$
    – Henry
    Mar 22 at 17:35
0
$\begingroup$

In a very simplified way:

Posterior probability = $P(\gamma | D)$ = probability that your parameter (or vector of parameters) $\gamma$ is equal to the value you've sampled given your dataset $D$.

Prior = arbitrary guess of the value of $\gamma$ based on an expert knowledge (or ignorance for uninformative prior)

However, a question you don't ask is "what means the likelihood" in the Bayesian formula: basically, it is the opposite of the posterior probability.

Likelihood = $P(D|\gamma)$ = probability that your data is observed for the given values of the parameter(s) $\gamma$

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.