0
$\begingroup$

Let $X \sim N(\theta, \sigma^2)$ where $\sigma^2$ is known. Let the prior density $\pi(\theta) =1, \theta \in \mathbb{R}$ to be the improper uniform density over the real line. Find the posterior distribution, $\pi(\theta \mid x)$ and posterior mean.

Suppose we denote the sampling distribution as $f(x \mid \theta)$. I know that $$\pi(\theta \mid x) = \frac{f(x \mid \theta)\pi(\theta)}{\int\limits f(x \mid \theta)\pi(\theta)d\theta}.$$ In this particular scenario, it would follow that $\pi(\theta \mid x) = f(x \mid \theta)$. I am confused as to how to get this sampling distribution. In my textbook, there is an example where $n$ iid samples are drawn from a $Bernoulli(p)$ distribution where the prior distribution on $p$ is $beta(\alpha, \beta)$. In that example, they utilized the fact that the sum of the $X_i$'s was $Binomial(n, p)$. However, in my example of an improper prior, even if I take the sum of the data, the distribution would be normal with the mean relying on $\theta$. But in this case, there's nothing I can use about $\theta$. Am I supposed to just use the likelihood function in this case to determine the sampling distribution? If that is the case, can someone help me explain why in some cases we take a sum of the data and other times not?

$\endgroup$
  • $\begingroup$ The likelihood function is a density in the sample $x$, ie integrates to one in $x$, not in the parameter $\theta$. $\endgroup$ – Xi'an Sep 22 at 5:30
1
$\begingroup$

Actually, $\pi(\theta | x ) = \frac{f(x|\theta)\pi(\theta)}{\int_{\Theta}f(x|\theta)\pi(\theta)d\theta} = \frac{f(x|\theta)1}{\int_{\Theta}f(x|\theta)1d\theta}\propto f(x|\theta)$. I suspect you are uncertain with the idea of a likelihood function.

The likelihood function $f(x|\theta)$ (or $f(\theta;x)$), is a function of the parameter $\theta$. Taking the Binomial example, suppose you have $f(\theta; x=4, n=6)$, then for any $\theta \in [0,1]$, say $\theta=0.3$, $f(\theta)=f(0.3)$ returns you the probability of observing four heads given six flips if your coin was biased with 0.3 probability of heads.Try to plot this for $\theta=0.01, 0.02, \dots, 0.98, 0.99$.

To reiterate, the likelihood function of your parameter. It takes the form of your sampling distribution, except you hold your data constant and take parameter as input instead.

To derive a posterior with improper priors, it is sufficient the integral $\int_\Theta f(\theta;x)\pi(\theta)d\theta < \infty$ (is finite).

See Bayesian Choice - Section 1.5

$\endgroup$
  • $\begingroup$ So, to clarify, all you need to do with an improper prior is to normalize the likelihood function. I.e. integrate it out over the parameter space (instead of over the possible outcomes) and divide the likelihood by it. However, it is possible that the posterior is improper and the divisor would turn out to be infinite or zero. For the binomial you happen to get a beta-distribution, but e.g. when you observe 0 heads and 0 tails, you have a problem (a Beta(0,0) is not a valid distribution). $\endgroup$ – Björn Sep 21 at 6:55
  • $\begingroup$ In your specific case of an improper prior that is uniform across the real line (ie. prior is constant = 1), yes that is all you do: normalize the likelihood function. I am not sure what you would do with an improper posterior, it is a bit beyond me at this point. But chapter 6.6.4 of bayesian choice discusses this briefly. Regarding 0 heads 0 tails example, I am not sure how you arrived at this? That means you have no data. Also, posterior distributions don't necessarily belong to certain families of distributions. u thinking conjugate priors ? $\endgroup$ – silly.deer Sep 21 at 7:04
  • $\begingroup$ Well, with no prior, you automatically get a conjugate posterior, if you get a proper prior, at all. In a way, that's why it's conjugate. With an improper posterior, the answer is usually: you need to put in some information (or switch to doing something exact in a frequentist way or something else). Usually, using an improper prior is just a bad idea - people have quite a few times not realized that they were happily sampling away from an improper posterior using Gibbs sampling and just did not notice (and published in journals etc.). $\endgroup$ – Björn Sep 22 at 7:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.