2
$\begingroup$

Suppose $X_1,X_2,...,X_n$ be iid random variables with $N(\mu,\sigma^2)$ distribution. We know that $X_i-\mu$ has a $N(0,\sigma^2)$ distribution.

My question is what is the distribution for $X_i-\bar{X}$? where ($\bar{X}=\frac{\Sigma_1^n X_i}{n}$)

I try to do as below. Since $X_i\sim N(\mu, \sigma^2)$ and $\bar{X}\sim N(\mu,\frac{\sigma^2}{n})$

So $X_i-\bar{X}\sim N(0,(1+\frac{1}{n})\sigma^2)$,however, $X_i$ and $\bar{X}$ are not independent, I think this simple solution is not correct.

Thank you very much.

$\endgroup$
  • 2
    $\begingroup$ Which part of the sample mean is independent of $X_1$? Which part not? $\endgroup$ – Michael M May 22 '16 at 12:03
  • 1
    $\begingroup$ If I understand right $\frac{X_2+X_3+...+X_n}{n}$ is independent of $X_1$, $\frac{X_1}{n}$ is not $\endgroup$ – Deep North May 22 '16 at 12:43
  • 1
    $\begingroup$ What do you know about variances of sums or differences of correlated random variables? $\endgroup$ – Glen_b May 23 '16 at 6:12
3
$\begingroup$

Updated with full solution since OP has now solved it.


For a complete solution, one needs to first show that $ Y_i:= X_i - \bar{X}$ is a Gaussian random variable, whence it suffices to find its mean and variance to characterize the distribution. Knowing something about Gaussian random vectors makes this straight-forward. This solution is presented first, and then I also provide a more direct argument without use of multivariate techniques.

Multivariate solution: If $X_i\overset{iid}{\sim} N(\mu, \sigma^2)$, then $\mathbf{x}: = [X_1, \dots, X_n]' \sim N_n(1_n\mu, \sigma^2I_n)$. We also know that for any conformable matrix $A$, $A'\mathbf{x}$ is Gaussian with mean $A'1_n\mu$ and covariance matrix $\sigma^2A'A$. Consider, without loss of generality, the case $i = 1$. Then we have $$A=[1 - 1/n, -1/n,\dots, -1/n]'.$$ That is, $Y_1$ is Gaussian with mean $A'1_n \mu = 0$ and variance $$\sigma^2 A'A = \sigma^2([1-1/n]^2+(n-1)/n^2) = \sigma^2(n-1)/n.$$ This completes the first solution.


Univariate solution: We can write $Y_i = (1-1/n)X_i - \sum_{j\neq i}X_j/n$, where the first term is independent of the second because functions of independent random variables are independent. We also know that sums of independent Gaussian random variables are still Gaussian, and that multiplying a Gaussian random variable by a constant gives another Gaussian random variable. Thus, using standard rules of means and variances, $$(1 - 1/n)X_i\sim N(0, (1-1/n)^2 \sigma^2)$$ and

$$ -1\sum_{j\neq i}X_j \sim N(0, (n-1)\sigma^2/n^2), $$

which implies that $$Y_i \sim N(0, \sigma^2(n-1)/n),$$

where we have used that independence implies zero covariance.

$\endgroup$
  • $\begingroup$ Thanks, should it be $A=[1 - 1/n, 1/n,\dots, 1/n]'$? $\endgroup$ – Deep North May 22 '16 at 13:24
  • $\begingroup$ No, I don't think so. Try the case $n = 2$. $[1 - 1/n, -1/n][X_1, X_2]' = X_1 - X_1/n - X_2/n = X_1 - (X_1 + X_2)/n = X_1 - \bar{X}$ $\endgroup$ – ekvall May 22 '16 at 13:27
  • $\begingroup$ I think I also find another solution without using Matrix. I will try to post later. $\endgroup$ – Deep North May 22 '16 at 13:31
2
$\begingroup$

Staying in the univarite case, since $X_i, i=1$ ,$\dots ,N$ are iid Normally distributed with mean $\mu$ and variance $\sigma^2$, we have, as you mentioned,

$E[X_i -\bar X] = E[X_i- \frac{1}{n}\sum_j^N X_j] = E[X_i] - \frac{1}{n}\sum_j^N E[X_j] = \mu - \frac{1}{n}n\mu =0$

For the variance, notice that

$Var[X_i-\bar X] = Var[X_i] + Var[\bar{X}] - 2Cov(X_i, \bar{X})$

Where

$Cov(X_i, \bar{X}) = Cov(X_i, \frac{1}{n}\sum_j^N X_j) = Cov(X_i, \frac{1}{n}X_i)$ by independence.

This should lead you to your answer.

Also, you technically have to check that the resulting distribution is itself normal. Then, using the shortcut argument that the normal distribution is entirely determined by its first two moments, we have that:

$X_i - \bar X \sim N \left (E[X_i - \bar X], Var[X_i - \bar X] \right )$

$\endgroup$
  • 2
    $\begingroup$ Welcome to the site! Please note that there is a reason these calculations were left out from my answer: this is a self-study question. You may read about it here: stats.stackexchange.com/tags/self-study/info $\endgroup$ – ekvall May 22 '16 at 14:05
  • 2
    $\begingroup$ Oh I see, thank you. I edited my solution so that it does not provide the answer. However, I think that pedagogically, going to the multivariate case is overkilling it so maybe OP can still benefit from this. $\endgroup$ – wiwh May 22 '16 at 14:14
  • $\begingroup$ I indeed missed a part. It is now orrected, thanks. Student001's answer is complete imho, especially with the justification of why we end up with a Gaussian RV. $\endgroup$ – wiwh May 24 '16 at 7:28
1
$\begingroup$

Student001 already gave a solution for the question and accepted. Micheal M's comments are also very helpful, so I will post another solution following Micheal's suggestion without using matrix notation.

$X_1-\bar{X}=X_1-\frac{X_1+X_2+...+X_n}{n}\\=X_1-\frac{X_1}{n}-\frac{X_2+X_3+...+X_n}{n}\\=(1-\frac{1}{n})X_1-\frac{1}{n}(X_2+X_3+...+X_n)$

Next we know that:

$(1-\frac{1}{n})X_1\sim N(\frac{n-1}{n}\mu,\frac{(n-1)^2}{n^2}\sigma^2) \tag 1$

$\frac{1}{n}(X_2+X_3+...+X_n)\sim \frac{1}{n}N((n-1)\mu,(n-1)\sigma^2)=N(\frac{n-1}{n},\frac{n-1}{n^2}\sigma^2) \tag 2$

And $(1)-(2)$ has a $N(0,\frac{(n-1)^2+n-1}{n^2}\sigma^2)=N(0,\frac{n-1}{n}\sigma^2)$

i.e $X_1-\bar{X}\sim N(0,\frac{n-1}{n}\sigma^2)$

This result is exactly the same as Student001's results.


Another method as suggest by Glen_b: we need to find the variance of $X_i$ and $\bar{X}$

$Var(X_i-\bar{X})=Var(X_i)+Var(\bar{X})-2Cov(X_i,\bar{X})$

The key is to calculate the $Cov(X_i,\bar{X})$

$Cov(X_i,\bar{X})=Cov(X_i, \frac{1}{n}(X_1+X_2+...+X_i+...+X_n))$

We will use the formula: $Cov(X,Y+Z)=Cov(X,Y)+Cov(X,Z)$

$Cov(X_i,\frac{1}{n}(X_1+X_2+...+X_i+...+X_n)=cov(X_i,\frac{1}{n}X_1)+...+Cov(X_i,\frac{1}{n}X_i)+...+Cov(X_i,\frac{1}{n}X_n)$

By i.i.d we know that except $Cov(X_i,\frac{1}{n}X_i)$ all other terms are zeros.

$\therefore Cov(X_i,\bar{X})=Cov(X_i,\frac{1}{n}X_i)=\frac{1}{n}Cov(X_i,X_i)=\frac{1}{n}Var(X_i)=\frac{1}{n}\sigma^2$

Finally,

$Var(X_i-\bar{X})=Var(X_i)+Var(\bar{X})-2Cov(X_i,\bar{X})=\sigma^2+\frac{\sigma^2}{n}-2\frac{1}{n}\sigma^2=\frac{n-1}{n}\sigma^2$

All methods get same results.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.