What is Bayesian posterior probability and how is it different to just using a p-value? I've got an extremely limited understanding of statistics, and every explanation I've found uses further technical terms that I don't understand. I'm trying to figure out:


*

*What exactly is posterior probability

*What exactly is a p-value (my understanding of this is that if your p-value is beneath your cut off then it suggests that your observation was not by chance, and the smaller your p-value is, the less it might be the case that it was chance. But I don't actually know what a p-value is)

*Are they interchangeable? What would be the difference between using one or the other? Are they suited to different things specifically?


Apologies for asking such a basic question, I'm really confused and no amount of basic explanations are actually basic enough.
 A: Maybe it would be easiest to see the differences in an example where the Bayesian answer doesn't at all look like the Frequentist answer.
Imagine a world where the American Society of Magicians convinces Congress to create magical quarters.  They are either 1/3rd biased for heads, 2/3rds biased for heads or a fair coin.  You want to test what kind of coin you have from a randomly selected coin from a bank.
For some reason, you decide that you are going to flip the coin exactly eight times and try to infer the true value from the coin toss.  You are going to use both a Bayesian and a Frequentist method to perform the test.
For the Frequentist test, you have three possible hypotheses in the set $\{\pi=1/3,\pi=1/2,\pi=2/3\}$.  You can only choose one, so you choose $\pi=1/3$.  If you had chosen a different null hypothesis, then you would get a different test solution.
Each test is a Bernoulli trial and the long run distribution will be the binomial distribution.  Since your null hypothesis is $\pi=1/3$ your test distribution will be in the following table.

Because there are only going to be 8 tosses and there is low power, we are only going to use a 70% degree of confidence.  The green area is the confidence region one would expect to see if $\pi=1/3$ and the yellow is the rejection region.  If you would repeat the test of eight tosses an infinite number of times and if the null is true then you should expect to see this distribution of toss frequencies.
Notice though that if the number of successes is zero, then not only would this reject the null of $\pi=1/3$ it would reject the others as well.  It is possible for a Frequentist solution to generate a rejection of all possible answers in an extreme case.
The p-value is the probability you would expect to see if the null is true.  If the null is not true, then we know nothing about the probabilities.   You are asserting it is true with the goal of falsifying it.
For the Bayesian problem, you do not choose a hypothesis.  You do choose a prior.  At first, you consider all three hypotheses as equally likely so you assign equal weight to each.  You realize this is a poor choice as there is probably data on how many of each are minted.  You go to the Mint's website and find out that the true probability for a randomly selected coin is 25% that $\pi=1/3$, 50% that $\pi=1/2$ and 25% otherwise.
The Frequentist solution does not depend on the actual data but on the hypothesis.  The Bayesian posterior depends upon the actual data that was collected so you can't create a test table until after you see the data unless you would calculate it for all possible cases.
We are going to assume that you saw six heads.  You take the prior and multiply it by the likelihood function, which in this case is the binomial, which gives you the product which is the numerator.  You sum all possible hypotheses as your normalization constant and divide each hypothesis by it to form your posterior mass function.  It isn't a density because it isn't a continuous variable.
The test distribution is in the chart below.

Notice that the solutions are different.  This isn't merely because of the prior, they will usually be different.  In fact, it isn't uncommon for something to be statistically significant with a Frequentist test and not with a Bayesian test.
The meaning of the p-value is that if your hypothesis is true, then the p-value is the probability of seeing a six, seven or eight by chance alone.  By fixing the parameter and treating the data as random, you can reduce the problem to a problem of chance.  If you reject the null, it really doesn't mean the null is false, it just means that the observation is very improbable if the null is true so it is probably false.  The big issue is that the concern is about the falseness of something and not the truth.  Your p-value is p<1.71%.  The less than sign is present because this is also a minimax distribution.
Because you are assuming there is no valuable prior knowledge, you need to minimize the maximum loss that could be created by acting on the knowledge created by the data.  Frequentist p-values are not true probability distributions, but rather worst-case distributions, if the null is true.
The meaning of the Bayesian posterior is that given the actual result, the probability that $\pi=1/3$ is 3.35%.  It is unlikely that it is the true value, but there is a small chance it is the actual value.  Notice though that if you had not called the Mint, then your posterior probability would have been 4.47%.  That is the subjective component.  Different people with different information or prior knowledge will get different results.
Because of this, they are not interchangeable.  In the Frequentist conceptualization, the data is random but the parameter is fixed when you choose a null.  In the Bayesian conceptualization, the data is not random, it is fixed because you saw it, but the parameter is uncertain.  In Bayesian thinking uncertainty and randomness is the same thing, while in the Frequentist thought chance and randomness are the same things.
There is one last difference.  The Frequentist and the Maximum Likelihood estimator is $\hat{\pi}=6/8=.75,$ which is not possible.  Another aspect of Frequentist estimators is that the result can be physically impossible.  The Bayesian result though is the entire posterior mass function.  Instead of a point, you get three possible answers with attached probabilities.  You can get a point estimator by assuming there is some cost to being wrong.  Without going into the math, the appropriate cost function is called the all-or-nothing cost function.  The most probable answer would be the one you use, which in this case is that $\pi=2/3$.  It can never happen that a Bayesian solution cannot happen.  It is a biased estimator though.  It is biased by the prior.  As a strong rule of thumb, all Bayesian solutions are biased.
A: 1) A posterior probability distribution is simply the probability distribution of your parameter after having seen the data. In the general framework of Bayesian statistics, you put a prior distribution on your parameter that reflects your belief about what it might be. You also have a likelihood function for your data that says if we know what the parameter is, then this is how the data will be distributed. Given these pieces of information along with the data we observe, we can determine the posterior probability distribution. The idea is that we have an initial belief about the parameter, then we observe some data that tells us something about the parameter, and then we update our belief based on the data. The result of this update is the posterior probability distribution. The posterior probability distribution tells us the degree of belief we should have for any particular value of the parameter. 
2) A p-value is a frequentist construction. A p-value tells us "What is the probability we observed this data if the null hypothesis is true?". This is why a low p-value implies it is less likely the null hypothesis is true. 
3) No, they're not interchangeable. The fundamental difference between a p-value and a posterior probability is that a p-value is a statement about the probability of observing data, while a posterior probability is a statement about the degree of belief of a particular parameter.
The difference in using one or the other is often a subjective choice. Some people like the frequentist philosophy better than the Bayesian philosophy and vice versa. One reason for using the Bayesian method is that you have the ability to specify a prior, which might be useful if you have domain information you wish to incorporate into your analysis. Bayesian methods can also be more effective in the case of small sample size, since they do not rely on asymptotics or large sample properties.  
