How does an observation condition the next one, if the numbers are exp. distributued with uknown average?

Question

We have a process that generates exponentially distributed random numbers, i.e., $P(X=x) = \lambda e^{-\lambda x}$. However, we don't know the value of $\lambda$. We observe the first realization with a value of $x_1$. The second, $x_2$ has not yet been realized. $x_1$ and $x_2$ are assumed independent.

The question is: what is the predictive distribution $P(x_2 | x_1)$? Both frequentist and Bayesian results are welcome.

(Extra: which estimation would perform better? the frequentist or the Bayesian?)

Answer 1

The improper prior on $\lambda$ selected by @Caprikuarius puts essentially all its weight on extremely large values of $\lambda$, which can, naturally, lead to poor performance when there's not much data.

Let's use Jeffreys' prior for the Exponential distribution instead: $p(\lambda) \propto 1/\lambda$. It has the advantage of being invariant to transforms of the parameter, whereas the Uniform distribution, proper or otherwise, is not. To see this, think about what the prior would be if we parameterized the Exponential distribution by its mean $\theta = 1/\lambda$. Would it still be Uniform over $(0,\infty)$ after the change of variable? Does this imply that a Uniform distribution on $\lambda$ is informative about $\theta$?

Discussions of what actually is an uninformative prior aside, writing the expression for the full posterior predictive distribution gets us:

$$p(x_2|x_1)\propto \int_0^{\infty}p(x_1,x_2|\lambda)p(\lambda)d\lambda$$

Since $x_1$ and $x_2$ are independent, we have $p(x_1,x_2|\lambda) = \lambda^2 e^{-(x_1+x_2)\lambda}$. Substituting results in:

$$p(x_2|x_1) \propto \int_0^{\infty}\lambda e^{-\lambda(x_1+x_2)}d\lambda = {1 \over (x_1+x_2)^2}$$

We can make the distributional form a little clearer (relative to standard forms) by dividing the denominator by the constant $x_1^2$:

$$p(x_2|x_1) \propto {1 \over (1+x_2/x_1)^2}$$

which is a Lomax distribution with $\alpha = 1$ and $\lambda = x_1$ (this $\lambda$ is the scale parameter used in the Wikipedia link.) The posterior mean of $x_2|x_1$ is just $x_1$, as one might hope (this is not true of the Uniform prior formulation, for which the posterior is a Lomax with shape parameter $\alpha=2$, scale parameter $x_1$, and posterior mean $x_1/2$,) but the variance is undefined for $\alpha \leq 2$.

Note that the posterior mean, in this case, is the same as the MLE. However, in the example constructed by the OP, we plug the MLE into the Exponential distribution in order to get an estimated distribution for $x_2$ (in fact, the MLE of the estimated distribution), whereas here we plug it into the Lomax distribution. The difference is that the latter accounts for uncertainty about $\lambda$, both due to the prior and the sample, whereas the former de facto assumes that the MLE is correct.

The reason the posterior mean in the Uniform case is $x_1/2$ instead of the possibly more desirable $x_1$ is the relatively large weight the improper Uniform prior puts on large values of $\lambda$ compared to the improper prior $1/\lambda$. Large values of $\lambda$ correspond to small expected values of $x_2$, hence the effect. Whether or not this is desirable is up to the user, of course!

Answer 2

Let's derive this step by step:

First of all, according to the Bayes Rule: $P(x_2 | x_1) = \frac{P(x_2 \cap x_1)}{P(x_1)}$

Since the $\lambda$ is unknown, we shall use the law of total probability:
$P(x_2 \cap x_1) = \int_{0}^{\infty}{P(x_2 \cap x_1 | \lambda) \cdot P(\lambda) d\lambda}$
$P(x_1) = \int_{0}^{\infty}{P(x_1 | \lambda) \cdot P(\lambda) d\lambda}$

Now, $P(x_2 \cap x_1)$ becomes $\frac{\int_{0}^{\infty}{P(x_2 \cap x_1 | \lambda) \cdot P(\lambda) d\lambda}}{\int_{0}^{\infty}{P(x_1 | \lambda) \cdot P(\lambda) d\lambda}}$

Since we assume no prior information on $\lambda$, it means we have equal chance to get any value of $\lambda$ in $[0, \infty)$, hence, $P(\lambda)$ becomes a constant, and we can cancel it out from the fraction.

Now, $P(x_2 \cap x_1)$ becomes $\frac{\int_{0}^{\infty}{P(x_2 \cap x_1 | \lambda) d\lambda}}{\int_{0}^{\infty}{P(x_1 | \lambda) d\lambda}}$

Notice that when $\lambda$ is fixed, $x_1$ and $x_2$ are independent, so $P(x_2 \cap x_1 | \lambda)$ = $P(x_2 | \lambda) \cdot P(x_1 | \lambda)$ = $(\lambda e^{- \lambda x_2}) \cdot (\lambda e^{- \lambda x_1}) = \lambda^2 e^{-\lambda (x_1 + x_2)}$

Evaluate the integrals: $P(x_1 | \lambda) = \int_{0}^{\infty}{\lambda e^{-\lambda x_1} d\lambda} = \frac{1}{x_1^2}$
$P(x_2 \cap x_1 | \lambda) = \int_{0}^{\infty}{\lambda^2 e^{-\lambda (x_1 + x_2)} d\lambda} = \frac{2}{(x_1 + x_2)^3}$

Finally, combining them all together, $P(x_2 | x_1) = \frac{P(x_2 \cap x_1)}{P(x_1)} = \frac{2x_1^2}{(x_1 + x_2)^3}$