We have a process that generates exponentially distributed random numbers, i.e., $P(X=x) = \lambda e^{-\lambda x}$. However, we don't know the value of $\lambda$. We observe the first realization with a value of $x_1$. The second, $x_2$ has not yet been realized. $x_1$ and $x_2$ are assumed independent.

The question is: how does the knowledge of the value $x_1$ of the first observation condition our prediction for the value of the next one, $x_2$? Specifically, what does $P(x_2 | x_1, ``X\textrm{ is exp."})$ look like? Both frequentist and Bayesian (assuming no prior information) results are welcome.

(Extra: which estimation would perform better? the frequentist or the Bayesian?)

UPDATE: The answer given by @Caprikuarius is $P(x_2 | x_1) = \frac{2x_1^2}{(x_1 + x_2)^3}$.

An alternative option is based on the maximum likelyhood estimation (MLE) of $\lambda$, denoted as $\hat{\lambda}$ . To find $\hat{\lambda}$, we look for the value of $\lambda$ that maximizes the probability of the observed value $x_1$. This results in $\hat{\lambda}=x_1^{-1}$. Then the probability of $x_2$ could be approximated by $P(x_2 | x_1) = \frac{1}{x_1}e^{-x_2/x_1}$. Although the MLE seems a very rough approximation, I have numerically compared the accuracy of the predictions made by the MLE and the Bayesian estimation, and the MLE is generally better (however, they become comparable when $x_1$ lies close to $\lambda^{-1}$).

Does anybody know why the MLE beats the Bayesian one? Of course, the Bayesian estimation is a heavy-tailed power-law so the predictions of $x_2$ will fluctuate much more than the exponential MLE, but beyond from that, why a seemingly more elaborated solution is beaten by a simple MLE? Shouldn't they be similar as far as we are using an uninformative prior, as is the case here?

  • $\begingroup$ Re extra: the performance of the Bayes estimator will depend on your prior. As far as the main question goes, are you assuming $x_1$ and $x_2$ are independent? If so, the answer is straightforward in either paradigm. If not, then what form of dependence are you assuming? $\endgroup$ – whuber Aug 30 '19 at 20:30
  • $\begingroup$ Yes, they are assumed, independent. Regarding the prior, I would use Jeffrey's scale prior between 0 and infinity since is the most uninformative one in this case. $\endgroup$ – DavidF Aug 30 '19 at 20:32
  • 1
    $\begingroup$ Wikipedia, under 'parameter estimation' has discussion of both frequentist (MLE, unbiased, and minimum MSE) estimates of $\lambda$ and Bayesian estimation using a conjugate beta prior. $\endgroup$ – BruceET Aug 30 '19 at 21:02
  • 1
    $\begingroup$ If $X_1$ and $X_2$ are as you describe, then why would they have different distributions? How would it make sense to use one observation $X_2$ to estimate $\lambda?$ // Are you familiar with the 'no-memory' property if exponential random variables? // If this is a textbook question, maybe you can tell us the context of the problem in text or lectures? If it is a a practical probability modeling problem, then maybe you can tell us more about what you are trying to do. // As the Question stands, it is not really clear (to me anyhow) what you're asking. $\endgroup$ – BruceET Aug 30 '19 at 21:50
  • 1
    $\begingroup$ @BruceET, I'm sorry but there's not more context. It is neither a textbook nor a practical problem. I just made up the problem and was curious about the solution. $\endgroup$ – DavidF Aug 30 '19 at 22:30

The improper prior on $\lambda$ selected by @Caprikuarius puts essentially all its weight on extremely large values of $\lambda$, which can, naturally, lead to poor performance when there's not much data.

Let's use Jeffreys' prior for the Exponential distribution instead: $p(\lambda) \propto 1/\lambda$. It has the advantage of being invariant to transforms of the parameter, whereas the Uniform distribution, proper or otherwise, is not. To see this, think about what the prior would be if we parameterized the Exponential distribution by its mean $\theta = 1/\lambda$. Would it still be Uniform over $(0,\infty)$ after the change of variable? Does this imply that a Uniform distribution on $\lambda$ is informative about $\theta$?

Discussions of what actually is an uninformative prior aside, writing the expression for the full posterior predictive distribution gets us:

$$p(x_2|x_1)\propto \int_0^{\infty}p(x_1,x_2|\lambda)p(\lambda)d\lambda$$

Since $x_1$ and $x_2$ are independent, we have $p(x_1,x_2|\lambda) = \lambda^2 e^{-(x_1+x_2)\lambda}$. Substituting results in:

$$p(x_2|x_1) \propto \int_0^{\infty}\lambda e^{-\lambda(x_1+x_2)}d\lambda = {1 \over (x_1+x_2)^2}$$

We can make the distributional form a little clearer (relative to standard forms) by dividing the denominator by the constant $x_1^2$:

$$p(x_2|x_1) \propto {1 \over (1+x_2/x_1)^2}$$

which is a Lomax distribution with $\alpha = 1$ and $\lambda = x_1$ (this $\lambda$ is the scale parameter used in the Wikipedia link.) The posterior mean of $x_2|x_1$ is just $x_1$, as one might hope (this is not true of the Uniform prior formulation, for which the posterior is a Lomax with shape parameter $\alpha=2$, scale parameter $x_1$, and posterior mean $x_1/2$,) but the variance is undefined for $\alpha \leq 2$.

Note that the posterior mean, in this case, is the same as the MLE. However, in the example constructed by the OP, we plug the MLE into the Exponential distribution in order to get an estimated distribution for $x_2$ (in fact, the MLE of the estimated distribution), whereas here we plug it into the Lomax distribution. The difference is that the latter accounts for uncertainty about $\lambda$, both due to the prior and the sample, whereas the former de facto assumes that the MLE is correct.

The reason the posterior mean in the Uniform case is $x_1/2$ instead of the possibly more desirable $x_1$ is the relatively large weight the improper Uniform prior puts on large values of $\lambda$ compared to the improper prior $1/\lambda$. Large values of $\lambda$ correspond to small expected values of $x_2$, hence the effect. Whether or not this is desirable is up to the user, of course!


Let's derive this step by step:

First of all, according to the Bayes Rule: $P(x_2 | x_1) = \frac{P(x_2 \cap x_1)}{P(x_1)}$

Since the $\lambda$ is unknown, we shall use the law of total probability:
$P(x_2 \cap x_1) = \int_{0}^{\infty}{P(x_2 \cap x_1 | \lambda) \cdot P(\lambda) d\lambda}$
$P(x_1) = \int_{0}^{\infty}{P(x_1 | \lambda) \cdot P(\lambda) d\lambda}$

Now, $P(x_2 \cap x_1)$ becomes $\frac{\int_{0}^{\infty}{P(x_2 \cap x_1 | \lambda) \cdot P(\lambda) d\lambda}}{\int_{0}^{\infty}{P(x_1 | \lambda) \cdot P(\lambda) d\lambda}}$

Since we assume no prior information on $\lambda$, it means we have equal chance to get any value of $\lambda$ in $[0, \infty)$, hence, $P(\lambda)$ becomes a constant, and we can cancel it out from the fraction.

Now, $P(x_2 \cap x_1)$ becomes $\frac{\int_{0}^{\infty}{P(x_2 \cap x_1 | \lambda) d\lambda}}{\int_{0}^{\infty}{P(x_1 | \lambda) d\lambda}}$

Notice that when $\lambda$ is fixed, $x_1$ and $x_2$ are independent, so $P(x_2 \cap x_1 | \lambda)$ = $P(x_2 | \lambda) \cdot P(x_1 | \lambda)$ = $(\lambda e^{- \lambda x_2}) \cdot (\lambda e^{- \lambda x_1}) = \lambda^2 e^{-\lambda (x_1 + x_2)}$

Evaluate the integrals: $P(x_1 | \lambda) = \int_{0}^{\infty}{\lambda e^{-\lambda x_1} d\lambda} = \frac{1}{x_1^2}$
$P(x_2 \cap x_1 | \lambda) = \int_{0}^{\infty}{\lambda^2 e^{-\lambda (x_1 + x_2)} d\lambda} = \frac{2}{(x_1 + x_2)^3}$

Finally, combining them all together, $P(x_2 | x_1) = \frac{P(x_2 \cap x_1)}{P(x_1)} = \frac{2x_1^2}{(x_1 + x_2)^3}$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.