Understanding Bayesian statistics

Question

OK, so I have no problem with the Bayes' theorem itself and I do think that I get a lot of the simpler maths around Bayesian statistics. However, I am struggling with the idea of treating population parameters as random variables.

Say someone tosses a coin. Before the toss, I estimate the probability to be 0.5. Coin lands, I don't know the result. My frequentist soul tells me that it is either heads or tails, but not a random variable with associated probability. In the long run, I can treat my estimate as the expected fraction of correct guesses, but for that one particular toss it is not applicable.

Our intuition is more Bayesian in the sense that even though the coin has landed, as long as we don't know the result, we still tend to think that the probability is 0.5 (we treat the reality of the coin toss as a random variable). However, this is relative: should someone get a peek at the coin and tell us: "I bet 50$ that it's a head!" we would not bet, would we? (in a way, it is the reverse of the famous xkcd strip).

I guess my question is naive, but here it comes: how do you deal with this paradox?

Answer 1

The coin example you refer to is one of the classic arguments for probabilistic reasoning.

Say someone tosses a coin. Before the toss, I estimate the probability to be 0.5. Coin lands, I don't know the result. My frequentist soul tells me that it is either heads or tails, but not a random variable with associated probability

You're correct in that the coin is either heads or tails. The probability that you assign it is a statement about your state of knowledge of the system, which contains uncertainty. This is particular to you: it is your reasoning that indicates to you, given the facts that you have in your possession (this is typically the $I$ in $p(\rm{heads} | I)$.

This is a fundamental point of difference, in probabilistic reasoning (or Bayesian reasoning), we try to recognize that the probability we discuss are not inherent features of the system under study (in this case a coin), but they are quantifiers of our (or someone robot's) state of knowledge about the system.

So the classic argument goes that someone else then 'takes a peek' at the coin. For them, the probability has collapsed to the coin being either heads or tails. Their state of knowledge now has no uncertainty. But for you, your state of knowledge remains unaffected by the fact that they looked. Now of course if they try to make a bet with you it may change your state of knowledge, but I'll leave my explanation at that.