# When is using Loss Function for posterior distribution necessary?

Let $$X_i\sim Poisson(\theta)$$ for $$i = 1, 2, ..., n.$$ Let $$\theta \sim Gamma(\alpha, \beta)$$ be the conjugate prior distribution.

It is easy to show that the posterior $$\theta|\textbf{x} \sim Gamma(n\bar{x} + \alpha,\beta+n),$$

If we already know how the posterior distribution behaves, do we need to use loss function to estimate the parameter? Can't we use $$\hat{\theta} = \frac{n\bar{x} + \hat{\alpha}}{\hat{\beta}+n}.$$ the estimated mean from the posterior distribution as the estimator for $$\theta$$?

• Why is the mean the estimate instead of the median or the mode or something else?
– mef
Jul 22, 2022 at 19:54
• @mef that is a good question and I just thought it is usual to assume $\hat{E}(\hat{\theta}) = \hat{\theta}$. It serves as a good approximation.
– RRMT
Jul 23, 2022 at 13:37
• "severs as a good approximation." According to what? In other words: according to your loss function!
– mef
Jul 23, 2022 at 14:24
• Yes, that's the point. For example, absolute error loss can produce the median.
– mef
Jul 23, 2022 at 14:33
• There is no easy answer. This is at the heart of decision theory. See for example Berger (1980) Statistical Decision Theory and Bayesian Analysis, 2nd ed.
– mef
Jul 23, 2022 at 14:46

do we need to use loss function to estimate the parameter?

An estimate is, in essence, a kind of summary. By choosing the expectation of the posterior as your summary, you've implicitly decided to minimize the squared error. This is because the expected value is the minimizer of the squared error.

But nothing prevented you from using the median, which would minimize a different loss function. The result from a Bayesian analysis is not a number, it is a distribution. How you decide to summarize that distribution will depend on what qualities you want your summary to have.

To auto-quote from The Bayesian Choice (2001),

Considering that the overall purpose of most inferential studies is to provide the statistician (or a client) with a decision, it seems reasonable to ask for an evaluation criterion of decision procedures that assesses the consequences of each decision and depends on the parameters of the model, i.e., the true state of the world (or of Nature).

These decisions can be of various kinds (...) and also include assessing whether a new scientific theory is compatible with the experimental evidence at hand. If no evaluation criterion is available, it is impossible to compare different decision procedures and absurd solutions, such as proposing $$\hat\theta = 3$$ for any real estimation problem or even more dramatically the answer one wants to impose, can only be eliminated by ad-hoc reasoning. To avoid such reasoning implies a reinforced axiomatization of the statistical inferential framework, called decision theory.

Although almost everybody agrees on the need for such an evaluation criterion, there is an important controversy running about the choice of this evaluation criterion, since the consequences on the decision are not innocuous. This difficulty even led some statisticians to totally reject decision theory, on the basis that a practical determination of the decision-maker evaluation criterion is utterly impossible in most cases.

This criterion is usually called loss and is defined as follows, where $$\mathcal D$$ denotes the set of possible decisions. $$\mathcal D$$ is called the {\it decision} space and most theoretical examples focus on the case $$\mathcal D = \Theta$$, which represents the standard estimation setting.

Definition 2.3 A loss function is any function $$\mathrm{L}$$ from $$\Theta \times {\mathcal D}$$ in $$[0,+\infty)$$.

This loss function is supposed to evaluate the penalty (or error) $$\mathrm L(\theta,d)$$ associated with the decision $$d$$ when the parameter takes the value $$\theta$$. In a traditional setting of parameter estimation, when $$\mathcal D$$ is $$\Theta$$ or $$h(\Theta)$$, the loss function $$\mathrm L(\theta,\delta)$$ measures the error made in evaluating $$h(\theta)$$ by $$\delta$$. Section 2.2 introduces a set of so-called rationality axioms that ensures the existence of such a function in a decision setting.

Further comments link to the dual nature of loss and prior stressed by Hermann Rubin (1987):

Many criticisms have been addressed on theoretical or psychological grounds against the notion of rationality of the decision-makers (...) First, it seems illusory to assume that individuals can compare all rewards, that is, that they can provide a total ordering of $$\mathcal P$$ because their discriminating abilities are necessarily limited, especially about contiguous or extreme alternatives. The transitivity assumption is also too strong, since examples in sports or politics show that real-life orderings of preferences often lead to nontransitivity, as illustrated by Condorcet and Simpson paradoxes (...) More fundamentally, the assumption that the ordering can be extended (...) has been strongly attacked because it implies that a social ordering can be derived from a set of individual orderings and this is not possible in general (see Arrow (1951) or Blyth (1993)). However, while recognizing this fact, Rubin (1987) notes that this impossibility just implies that utility and prior are not separable, not that an optimal (Bayesian) decision cannot be obtained, and he gives a restricted set of axioms pertaining to this purpose. In general, the criticisms above are obviously valuable, but cannot stand against the absolute need of an axiomatic framework validating decision-making under uncertainty.