9
$\begingroup$

We have N samples, $X_i$, from a uniform distribution $[0,\theta]$ where $\theta$ is unknown. Estimate $\theta$ from the data.

So, Bayes' rule...

$f(\theta | {X_i}) = \frac{f({X_i}|\theta)f(\theta)}{f({X_i})}$

and the likelihood is:

$f({X_i}|\theta) = \prod_{i=1}^N \frac{1}{\theta}$ (edit: when $0 \le X_i \le \theta$ for all $i$, and 0 otherwise -- thanks whuber)

but with no other information about $\theta$, it seems like the prior should be proportional to $1$ (i.e. uniform) or to $\frac{1}{L}$ (Jeffreys prior?) on $[0,\infty]$ but then my integrals don't converge, and I'm not sure how to proceed. Any ideas?

$\endgroup$
  • 1
    $\begingroup$ Your likelihood is incorrect: it will be zero whenever $\theta$ is less than the largest $X_i$. $\endgroup$ – whuber Sep 6 '13 at 15:40
  • $\begingroup$ Can you show what integrals you are taking? $\endgroup$ – user25658 Sep 6 '13 at 15:50
  • $\begingroup$ Yea, so, I guess I just don't know how to deal with the improper prior. E.g., I want to write $f[X_i] = \int_\Theta f(X_i|\theta)f(\theta)d\theta$ $\endgroup$ – Will Sep 6 '13 at 16:51
  • $\begingroup$ For the improper prior, $f[X_i] = \int_\Theta f(X_i|\theta)f(\theta)d\theta$ = $\int_{\max(X_i)}^\infty \theta^{-N}d\theta$ = $\max(X_i)^{1-N}/(N-1)$ and for the prior $f(\theta)\propto 1/\theta$ you similarly obtain $\max(X_i)^{-N}/N.$ Because $\max{X_i}\gt 0$ almost surely, it is certain the integrals will converge. $\endgroup$ – whuber Sep 10 '13 at 18:18
  • 1
    $\begingroup$ The Bernardo reference posterior is Pareto - see the catalog of noninformative priors. $\endgroup$ – Stéphane Laurent Sep 10 '13 at 20:30
3
$\begingroup$

This has generated some interesting debate, but note that it really doesn't make much difference to the question of interest. Personally I think that because $\theta$ is a scale parameter, the transformation group argument is appropriate, leading to a prior of

$$\begin{array}& p(\theta|I)=\frac{\theta^{-1}}{\log\left(\frac{U}{L}\right)}\propto\theta^{-1} & L<\theta<U\end{array}$$

This distribution has the same form under rescaling of the problem (the likelihood also remains "invariant" under rescaling). The kernel of this prior, $f(y)=y^{-1}$ can be derived by solving the functional equation $af(ay)=f(y)$. The values $L,U$ depend on the problem, and really only matter if the sample size is very small (like 1 or 2). The posterior is a truncated pareto, given by:

$$\begin{array}\\ p(\theta|DI)=\frac{N\theta^{-N-1}}{ (L^{*})^{-N}-U^{-N}} & L^{*}<\theta<U & \text{where} & L^{*}=max(L,X_{(N)}) \end{array}$$ Where $X_{(N)}$ is the Nth order statistic, or the maximum value of the sample. We get the posterior mean of $$E(\theta|DI)= \frac{ N((L^{*})^{1-N}-U^{1-N}) }{ (N-1)((L^{*})^{-N}-U^{-N}) }=\frac{N}{N-1}L^{*}\left(\frac{ 1-\left[\frac{L^{*}}{U}\right]^{N-1} }{ 1-\left[\frac{L^{*}}{U}\right]^{N} }\right)$$ If we set $U\to\infty$ and $L\to 0$ the we get the simpler exression $E(\theta|DI)=\frac{N}{N-1}X_{(N)}$.

But now suppose we use a more general prior, given by $p(\theta|cI)\propto\theta^{-c-1}$ (note that we keep the limits $L,U$ to ensure everything is proper - no singular maths then). The posterior is then the same as above, but with $N$ replaced by $c+N$ - provided that $c+N\geq 0$. Repeating the above calculations, we the simplified posterior mean of

$$E(\theta|DI)=\frac{N+c}{N+c-1}X_{(N)}$$

So the uniform prior ($c=-1$) will give an estimate of $\frac{N-1}{N-2}X_{(N)}$ provided that $N\geq 2$ (mean is infinite for $N=2$). This shows that the debate here is a bit like whether or not to use $N$ or $N-1$ as the divisor in the variance estimate.

One argument against the use of the improper uniform prior in this case is that the posterior is improper when $N=1$, as it is proportional to $\theta^{-1}$. But this only matters if $N=1$ or is very small.

$\endgroup$
1
$\begingroup$

Since the purpose here is presumably to obtain some valid and useful estimate of $\theta$, the prior distribution should be consistent with the specification of the distribution of the population from which the sample comes. This does NOT in any way mean that we "calculate" the prior using the sample itself -this would nullify the validity of the whole procedure. We do know that the population from which the sample comes is a population of i.i.d. uniform random variables each ranging in $[0,\theta]$. This is a maintained assumption and is part of the prior information that we possess (and it has nothing to do with the sample, i.e. with a specific realization of a subset of these random variables).

Now assume that this population consists of $m$ random variables, (while our sample consists of $n<m$ realizations of $n$ random variables). The maintained assumption tells us that $$\max_{i=1,...,n}\{X_i\}\le \max_{j=1,...,m}\{X_j\} \le \theta$$

Denote for compactness $\max_{i=1,...,n}\{X_i\} \equiv X^*$. Then we have $\theta \ge X^*$ which can also be written $$\theta = cX^*\qquad c\ge 1$$

The density function of the $\max$ of $N$ i.i.d Uniform r.v.'s ranging in $[0,\theta]$ is $$f_{X^*}(x^*) = N\frac {(x^*)^{N-1}}{\theta^N} $$

for the support $[0,\theta]$, and zero elsewhere. Then by using $\theta = cX^*$ and applying the change-of-variable formula we obtain a prior distribution for $\theta$ that is consistent with the maintained assumption: $$f_p(\theta) = N\frac {(\frac{\theta}{c})^{N-1}}{\theta^N}\frac 1c = \frac {N}{c^N} \theta^{-1}\qquad \theta \in [x^*, \infty]$$

which may be improper if we don't specify the constant $c$ suitably. But our interest lies in having a proper posterior for $\theta$, and also, we do not want to restrict the possible values of $\theta$ (beyond the restriction implied by the maintained assumption). So we leave $c$ undetermined.
Then writing $\mathbf X = \{x_1,..,x_n\}$ the posterior is

$$f(\theta \mid \mathbf X)\; \propto\; \theta^{-N}\frac {N}{c^N} \theta^{-1} \Rightarrow f(\theta \mid \mathbf X) = A\frac {N}{c^N} \theta^{-(N+1)}$$

for some normalizing constant A. We want $$\int_{S_{\theta}}f(\theta \mid \mathbf X)d\theta =1 \Rightarrow \int_{x^*}^{\infty}A\frac {N}{c^N} \theta^{-(N+1)}d\theta =1$$

$$\Rightarrow A\frac {N}{c^N}\frac {1}{-N}\theta^{-N}\Big |_{x^*}^{\infty} = 1 \Rightarrow A = (cx^*)^N$$

Inserting into the posterior $$f(\theta \mid \mathbf X) = (cx^*)^N\frac {N}{c^N} \theta^{-(N+1)} = N(x^*)^N\theta^{-(N+1)} $$

Note that the undetermined constant $c$ of the prior distribution has conveniently cancelled out.

The posterior summarizes all the information that the specific sample can give us regarding the value of $\theta$. If we want to obtain a specific value for $\theta$ we can easily calculate the expected value of the posterior, $$E(\theta\mid \mathbf X) = \int_{x^*}^{\infty}\theta N(x^*)^N\theta^{-(N+1)}d\theta = -\frac{N}{N-1}(x^*)^N\theta^{-N+1}\Big |_{x^*}^{\infty} = \frac{N}{N-1}x^*$$

Is there any intuition in this result? Well, as the number of $X$'s increases, the more likely is that the maximum realization among them will be closer and closer to their upper bound, $\theta$ - which is exactly what the posterior mean value of $\theta$ reflects: if, say, $N=2 \Rightarrow E(\theta\mid \mathbf X) = 2x^*$, but if $N=10 \Rightarrow E(\theta\mid \mathbf X) = \frac{10}{9}x^*$. This shows that our tactic regarding the selection of the prior was reasonable and consistent with the problem at hand, but not necessarily "optimal" in some sense.

$\endgroup$
  • $\begingroup$ Basing the prior on the data sounds fishy to me. How do you justify this approach? $\endgroup$ – whuber Sep 8 '13 at 1:23
  • $\begingroup$ I didn't base the prior on the data. I based the prior on the knowledge that the sample comes from a population of non-negative uniform random variables that have as an upper bound the unknown parameter. $\endgroup$ – Alecos Papadopoulos Sep 8 '13 at 2:50
  • 1
    $\begingroup$ I have nothing against the fact that your prior is not "the best". Where did I say something like that ? I'm just trying to understand your approach. I don't understand this equality yet. If $c$ is constant in the equality $\theta=cX^*$, does that mean that both $X^*$ and $\theta$ are nonrandom ? By the way you don't use the fact that $c \geq 1$ in the derivation of the prior, do you ? (cc @whuber) $\endgroup$ – Stéphane Laurent Sep 11 '13 at 6:14
  • 1
    $\begingroup$ And the support of your prior depends on the data ? ($\theta \in [x^*, \infty[$) $\endgroup$ – Stéphane Laurent Sep 11 '13 at 12:23
  • 2
    $\begingroup$ A prior depending (even if this is only through the support) on the data sounds wrong: you cannot know the maximum of the sample before the sample has been generated. Moreover, you claim that $\theta = cX^*$ is an almost sure equality, with both $\theta$ and $X^*$ random (thus there is correlation $1$). But this implies that the posterior distribution of $\theta$ (which is the conditional distribution of $\theta$ given the sample) is the Dirac mass at $cx^*$. And this contradicts your derivation of the posterior distribution. ... (no characters left...) $\endgroup$ – Stéphane Laurent Sep 11 '13 at 15:23
0
$\begingroup$

Uniform Prior Distribution Theorem (interval case):

"If the totality of Your information about $\theta$ external to the data $D$ is captured by the single proposition $$B=\{\{\text{Possible values for } \theta\}=\{\text{the interval } (a,b)\},a<b\}$$ then Your only possible logically-internally-consistent prior specification is $$f(\theta)=\text{Uniform}(a,b)$$

Thus, you prior specification should correspond to the Jeffrey's prior if you truly believe in the above theorem."

Not part of the Uniform Prior Distribution Theorem:

Alternatively you could specify your prior distribution $f(\theta)$ as a Pareto distribution, which is the conjugate distribution for the uniform, knowing that you posterior distribution will have to be another uniform distribution by conjugacy. However, if you use the Pareto distribution, then you will need to specify parameters of the Pareto distribution in some sort of way.

$\endgroup$
  • 4
    $\begingroup$ First you say the "only possible logically internally consistent" answer is a uniform distribution and then you proceed to propose an alternative. That sounds illogical and inconsistent to me :-). $\endgroup$ – whuber Sep 6 '13 at 16:11
  • 2
    $\begingroup$ I can't agree. For instance, $B$ is also the set $\{\theta | \theta^3\in(a^3, b^3)\}.$ When $\Theta\sim\text{Uniform}(a,b),$ the PDF of $\Psi=\Theta^3$ is $1/(3\psi^{2/3}(b-a))$ for $a^3\lt \psi\lt b^3$. But according to the "theorem," $\Psi\sim\text{Uniform}(a^3,b^3)$ whose pdf is $1/(b^3-a^3)$ in that interval. In short, although the proposition does not depend on how the problem is parameterized, the "theorem"'s conclusion does depend on the parameterization, whence it is ambiguous. $\endgroup$ – whuber Sep 6 '13 at 17:58
  • 2
    $\begingroup$ BabakP: How could one say this is a theorem ? A theorem is a mathematical claim with a mathematical proof. This "theorem" would be more appropriately termed as a "principle", but it is not sensible because it is contradictory, as shown by @whuber. $\endgroup$ – Stéphane Laurent Sep 10 '13 at 20:55
  • 2
    $\begingroup$ Thanks for the reference BabakP. I would like to point out that the "proof sketch" is bogus. Draper divides the interval into a finite number of equally spaced values and "passes to the limit." Anybody can divide the interval into values spaced to approximate any density they like and similarly pass to the limit, producing perfectly arbitrary "only possible logically-internally-consistent prior specifications." This kind of stuff--namely, using bad mathematics in an effort to show that non-Bayesians are illogical--gives Bayesian analysis an (undeservedly) bad name. (cc @Stéphane.) $\endgroup$ – whuber Sep 10 '13 at 21:11
  • 1
    $\begingroup$ @Stéphane Please forgive my insensitivity (insensibilité)--I admire your skill at interacting here in a second language and do not knowingly use obscure terms! Bogus is an adjective that comes from a 200-year old US slang term referring to a machine for counterfeiting money. In this case it's a mathematical machine for counterfeiting theorems :-). $\endgroup$ – whuber Sep 10 '13 at 21:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.