3
$\begingroup$

Suppose I draw $n$ samples $x_1...x_n$ of a random variable $x$, which is normally distributed with an unknown mean $\mu$ and variance $\sigma^2$. From those samples, I compute a sample mean $\bar x$ and a sample variance $s^2$. I wish to compute the distribution of $x_{n+1} \mid \bar x, s^2$, i.e. the distribution of one additional sample $x_{n+1}$, given my measured $\bar x$ and $s^2$.

How should I proceed? This is probably a bad way of proceeding with loads of issues, but here is my thought process so far:

  1. Start with the fact that $\frac{\bar{x} - \mu}{s/\sqrt{n}}$ is $t$-distributed, and then use that to find a likelihood of $\mu$ given $\bar x$.
  2. Use the fact that $\frac{x_{n+1} - \mu}{\sigma} \sim \mathcal N(0,1)$.
  3. Use the fact that $\frac{(n-1)s^2}{\sigma^2} \sim \chi^2_{n-1}$
  4. Combine these facts to get a distribution of $x_{n+1}$.

An alternative approach I thought of goes as follows:

  1. $x_{n+1}\sim N(\mu, \sigma^2)$
  2. $\bar x \sim N(\mu, \frac{\sigma^2}{n})$
  3. $(x_{n+1} - \bar x) \sim N(0, \sigma^2 + \frac{\sigma^2}{n})$
  4. Find the distribution of $\sigma^2 \mid s^2$ using the fact that $\frac{(n-1)s^2}{\sigma^2} \sim \chi^2_{n-1}$
  5. Integrate $N(0, \sigma^2 + \frac{\sigma^2}{n})$ over all possible $\sigma^2$ weighted by the distribution of $\sigma^2 \mid s^2$
$\endgroup$
9
  • 3
    $\begingroup$ Somewhere a dead monk named Bayes is turning in his grave at these hoops the Frequentists have made you jump through. $\endgroup$
    – bdeonovic
    Commented Oct 28, 2021 at 18:55
  • 4
    $\begingroup$ The point about Bayes is apt because the question as posed does not have a solution: the conditional distribution depends on the unknown parameters. The classical approach is to base testing or prediction on the estimated joint distribution of $(\bar x, x_{n+1}),$ while the Bayesian approach updates a prior distribution on the parameters--and gives the kind of answer you are looking for. $\endgroup$
    – whuber
    Commented Oct 28, 2021 at 19:07
  • 3
    $\begingroup$ oh no its not your fault @Danny, I'm just lamenting the state of statistical education $\endgroup$
    – bdeonovic
    Commented Oct 28, 2021 at 19:13
  • 1
    $\begingroup$ You don't have "samples" $x_1,\ldots,x_n$; rather you have a sample $x_1,\ldots,x_n.$ And $x_{n+1}$ is not one additional sample; it is one additional observation. $\endgroup$ Commented Oct 28, 2021 at 20:24
  • 2
    $\begingroup$ The word sample has a particular meaning in statistics. You claimed to be a lowly physicist but then attempt to correct a statistician on the meaning of the term in this statistical setting.... That's a risky strategy. en.wikipedia.org/wiki/Sample_(statistics) $\endgroup$
    – Glen_b
    Commented Oct 29, 2021 at 1:37

3 Answers 3

7
$\begingroup$

The distribution of $x_{n+1}$ conditional on $\overline x=(x_1 +\cdots +x_n)/n$ and $s^2 = \big((x_1-\overline x)^2+\cdots+(x_n-\overline x)^2\big)/(n-1)$ is $\operatorname N(\mu,\sigma^2),$ since the observations $x_1,\ldots,x_n,x_{n+1}$ are independent. Thus I suspect that what you actually want is the distribution of $(x_{n+1}-\overline x)/s,$ upon which you can base a prediction interval whose endpoints are $\overline x\pm c\cdot s_n,$ where $c$ is chosen so as to get a desired probability that $x_{n+1}$ is in the interval.

Notice that $x_{n+1}-\overline x\sim\operatorname N\left(0,\sigma^2\left( 1 + \frac 1 n \right)\right),$ and since $s^2$ is independent of $\overline x$ and of $x_{n+1},$ and $(n-1)s^2/\sigma^2\sim\chi^2_{n-1},$ you have $$ \frac{(x_{n+1}-\overline x)/\sqrt{1+\frac 1 n}}{s/\sqrt n} \sim t_{n-1}. $$ Thus if you choose $c$ so that $\Pr(-c<t_{n-1}<+c)= 1-\alpha,$ then you have $$ \Pr\left(x_{n+1} \text{ is between } \overline x \pm c\cdot\frac s {\sqrt n}\cdot\sqrt{1+\tfrac 1 n} \right) = 1-\alpha. $$ This is a prediction interval for $x_{n+1}.$

$\endgroup$
1
  • $\begingroup$ I think precisely, I want the distribution of $x_{n+1} - \overline x$ in terms of $s$ rather than $\sigma$. Maybe I'm thinking about it incorrectly, but either way, your answer gets me there, along with @Mang's. $\endgroup$ Commented Oct 28, 2021 at 23:30
3
$\begingroup$

As @whuber points out, this question is much more naturally answered in a Bayesian context. In that case the distribution you are interested in is known as the posterior predictive distribution.

$$ p(x_{n+1}|x_1,\ldots,n) = \int_\theta p(x_{n+1}|\theta,x_1,\ldots,x_n)p(\theta|x_1,\ldots,x_n)d\theta $$

where $\theta = (\mu,\sigma^2)$. This is about as close as you can get in the general case. If you are willing to accept a particular prior distribution for $\mu$ and $\sigma^2$ (Normal for $\mu$ and Inverse-Gamma for $\sigma^2$) you can get a closed form distribution (it is a $t$ distribution) which you can look up here

$\endgroup$
2
$\begingroup$

I believe that the following holds:

$$ T^*:= \sqrt{\frac{n}{n+1}}\frac{(X_{n+1}-\bar{X_n})}{s_n} \sim t_{n-1}$$

So if you define the test statistic $T^*$, you can thus derive confidence intervals, point estimates, and other quantities of interest with respect to $X_{n+1}$.

Proof:
First, note that $X_{n+1}, \bar{X_n}, s_n$ are all mutually independent.

Then, $\bar{X_n} \sim N(\mu, \frac{\sigma^2}{n})$, $X_{n+1} \sim N(\mu, \sigma^2)$, which implies (per your footnote) that $X_{n+1} - \bar{X_n} \sim N(0,\frac{n+1}{n} \sigma^2)$.

Thus, $$Z:= \frac{1}{\sigma} \sqrt{\frac{n}{n+1}} (X_{n+1} - \bar{X_n}) \sim N(0,1)$$

It's well known that $$V:= (n-1) \frac{s^2_n}{\sigma^2} \sim \chi^2_{n-1}$$ where $s^2_n = \frac{1}{n-1}\sum_{i=1}^n (X_i-\bar{X})$.

Since $Z$ and $V$ are independent, by definition of the t distribution, it follows that:

$$T^* = Z \sqrt{\frac{n-1}{V}} \sim t_{n-1}$$


This can be easily verified by simulation as well: Simulate n+1 independent normals using your favorite software, look at the density plot of your test statistic $T^*$ and compare with the $t_{n-1}$ distribution.

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.