2
$\begingroup$

My notes are rather light when it comes to this topic. I understand that the bayesian estimator, defined as (for sample space $\hat{x}$):

$E[\Theta | \hat{x}] = \int_{ \forall \Theta}yf_{\theta|\hat{x}}(y|\hat{x})dy $ (ie. the mean of the posterior distribution).

You can then look at loss functions where the form of the loss function determines that of $\theta^{'}$ (the estimator of $\theta$). By setting the loss function as a quadratic, absolute error or zero-one you get $\theta^{'}$ as the mean of the posterior, the median of the posterior and the mode of the posterior respectively.

The first two proofs make sense to me but it is the 3rd that I am not sure about. This is my understanding:

$E[l] = \int l(\theta^{'}, \theta)f(\theta|\hat{x})d\theta = \int (\theta^{'} - \theta)f(\theta|\hat{x})d\theta $

If $\theta^{'} = \theta \Rightarrow E[l] = 0$

If $\theta^{'} \neq \theta \Rightarrow \int (\theta^{'} - \theta)f(\theta|\hat{x})d\theta = 1 $ (as integrating over whole domain of $\theta$).

This proof obviously isn't rigorous, (and it could be incorrect too). What I can't see regardless is how this can be regarded as the 'mode' of the posterior distribution.

Thanks in advance!

$\endgroup$

migrated from math.stackexchange.com Feb 21 '18 at 22:52

This question came from our site for people studying math at any level and professionals in related fields.

3
$\begingroup$

You need to be a bit careful with this kind of problem because the definition of the zero-one loss function will depend on whether you are dealing with a discrete or continuous parameter. For a discrete parameter you can define the zero-one loss as an indicator function and this works fine. For a continuous parameter you can't do this, because if you integrate a discrete indicator over a continuous probability density function you will always get zero (so the expected loss would be zero regardless of the parameter estimator). In the latter case you need to define the zero-one loss function either by allowing some "tolerance" around the exact value, or by using the Dirac delta function. Below I show the derivation of the posterior mode estimator in both the discrete and continuous cases, using the Dirac function in the latter.


Discrete case: Suppose that the unknown parameter $\theta$ is a discrete random variable, and let $\hat{\theta}$ denote the estimator of this parameter. Then the zero-one loss function is defined as:

$$L(\hat{\theta} , \theta) = \mathbb{I}(\hat{\theta} \neq \theta).$$

This gives expected loss:

$$\begin{equation} \begin{aligned} \bar{L}(\hat{\theta} | X) \equiv \mathbb{E}(L(\hat{\theta}, \theta ) | X) &= \sum_{\theta \in \Theta} \mathbb{I}(\hat{\theta} \neq \theta) \pi (\theta | X ) \\[8pt] &= 1 - \sum_{\theta \in \Theta} \mathbb{I}(\hat{\theta} =\theta) \pi (\theta | X ) \\[8pt] &= 1 - \pi (\hat{\theta} | X). \end{aligned} \end{equation}$$

Minimising the expected loss is equivalent to maximising the posterior probability $\pi (\hat{\theta} | X)$, which occurs when $\hat{\theta}$ is the posterior mode.


Continuous case: Suppose that the unknown parameter $\theta$ is a continuous random variable, and let $\hat{\theta}$ denote the estimator of this parameter. Then the zero-one loss function is defined as:

$$L(\hat{\theta} , \theta) = 1 - \delta (\hat{\theta} - \theta).$$

where $\delta$ denotes the Dirac delta function. This gives expected loss:

$$\begin{equation} \begin{aligned} \bar{L}(\hat{\theta} | X) \equiv \mathbb{E}(L(\hat{\theta}, \theta ) | X) &= \int_{\Theta} (1- \delta (\hat{\theta} - \theta)) \pi (\theta | X ) d \theta \\[8pt] &= 1 - \int_{\Theta} \delta (\hat{\theta} =\theta) \pi (\theta | X ) d \theta \\[8pt] &= 1 - \pi (\hat{\theta} | X). \end{aligned} \end{equation}$$

Minimising the expected loss is equivalent to maximising the posterior density $\pi (\hat{\theta} | X)$, which occurs when $\hat{\theta}$ is the posterior mode.


$\endgroup$
  • $\begingroup$ Hey thank you for your answer! Just to be clear $\delta(\hat{\theta} - \theta) = \inf $ if $ \hat{\theta} - \theta = 0; 0 $ if $\hat{\theta} - \theta \neq 0$ $\endgroup$ – mrhappysmile Feb 22 '18 at 7:53
  • $\begingroup$ Kind of. The Dirac delta is not actually a "function" but it is usually likened to a function with those values and an integral of one; technically it is a distribution. $\endgroup$ – Ben Feb 22 '18 at 7:58
  • $\begingroup$ Thank you Ben, that makes sense. I appreciate your help! $\endgroup$ – mrhappysmile Feb 22 '18 at 8:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.