2
$\begingroup$

I am studying the differences in bayesian and frequentist approaches to point estimation.

I understand that there are objective and subjective approaches to Bayesian and some people don't like the subjective point of view.

Putting the subjectivity of choosing a prior aside (which I am not that uncomfortable with), I am mostly puzzled with the subjectivity of Bayesian risk.

Bayesian posterior risk is defined as follow

$R(\theta,\delta) = \int L(\theta,\delta(x))p(\theta|x)d\theta$

where posterior is computed with the Bayes formula.

Since the posterior is calculated starting from our subjective prior, our risk becomes subjective as well. Different priors yield different risks. This sounds strange because the real risk should be with respect to the loss function $L$, and choice of $\delta$; and it should be based on the true state of nature, not on our bliefs. Moreover this is not formulated as a relative risk of our belief with respect to the real risk. It is formulated as if we are dictating our belief as the real risk. This sounds to me more theological than scientific.

How do Bayesians reconcile this issue, or what is the rationale?

Thanks

$\endgroup$
  • 1
    $\begingroup$ If you replace "belief" with "knowledge" or "information" then I think some of the mystery disappears. If you knew the "true state of nature" then all risks would disappear as you would know the loss for each possible action and there would be zero risk. $\endgroup$ – Rasmus Bååth Apr 20 '15 at 8:59
  • 1
    $\begingroup$ As it happens, ET Jayes book is available online. I recomend the preface, and chapters 16 and 17 to help address your question. med.mcgill.ca/epidemiology/hanley/bios601/GaussianModel/… $\endgroup$ – D L Dahly Apr 20 '15 at 9:00
4
$\begingroup$

Bayes risk is a framework for making decisions. You identify a statistical model for your data and incorporate prior knowledge to obtain a posterior distribution via Bayes rule. To make a decision, you decide on a loss function L associated with a set of possible decisions. Your optimal decision, is the one that minimizes your Bayes risk, i.e. minimizes the loss function when integrated over your uncertainty.

Please notice that every step in the process is subjective including the model, the prior, the loss function, and the possible decisions.

Now, you correctly point out that your optimal decision might be extremely sub-optimal relative to the reality of the world. Unfortunately, if we have a poor model, prior, loss function, or set of possible decisions, a sub-optimal decision is likely. Of course, if we knew the "correct" model, prior, loss function, or set of possible decisions, we would use it.

$\endgroup$
  • $\begingroup$ I am upvoting this answer for the first paragraph, even though the second one is puzzling. It seems to me you could just as well replace "subjective" by "objective" and create a demonstrably true (and useful) statement, "every step in the process is objective including the (choices of a) model, the prior, the loss function, and the possible decisions." Could you explain the sense in which you find these to be subjective? $\endgroup$ – whuber Apr 20 '15 at 18:38
0
$\begingroup$

To reconcile this issue, building on what Jaradniemi said, consider how the posterior and loss (or personal utility) functions are used in Decision Analysis. The early book by H. Raiffa still applies, but an eloquent and well argued intro is R. Howard "The Foundations of Decision Analysis Revisited" Whereas a statistician is attempting to make a generally applicable statement about the state of nature, the decision analyst is "engineering" the best decision-policy based on the current state of knowledge, for the individual who has the decision to make.

$\endgroup$
0
$\begingroup$

Since the posterior is calculated starting from our subjective prior, our risk becomes subjective as well.

When you observe the data, and update the probabilities, the prior's impact becomes smaller and smaller. If you have a very large data set, you can start with any prior and will end up with the same posterior. So, the reconciliation is in the convergence of the posterior to what comes from the data.

Here's a simple analogy. Let's say, you're computing exponential weighted average: $$\mu_t=(1-\gamma)x_t+\gamma\mu_{t-1}$$. At time $t=0$ when you have no observation, you guess that the mean must be one: $\mu_0=1$, then you start updating: $$\mu_1=(1-\gamma)x_1+\gamma \mu_0$$ $$\mu_2=(1-\gamma)x_2+\gamma \mu_1=(1-\gamma)x_2+\gamma(1-\gamma)x_1+\gamma^2 \mu_0$$ etc. You can see that at time $t$ the weight on the initial guess $\mu_0$ is $\gamma^t$, i.e. its impact is vanishing.

It's a similar process with Bayesian risk, when your prior's influence on posterior vanishes as the sample grows.

On a small sample, of course, the impact is huge. However, on a small sample you could argue that non-Bayesian (say frequentist) approach is also unreliable due to large sample variances. If you're in a camp which says "let the data speak for itself", then small sample is an issue, while in large sample you get Bayesian approach to render the same result. So, on a small sample Bayesian approach "adds" to the data its subjective opinion, and it might be just what is needed.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.