# What is the distribution of $X_i-\bar{X}$ when $X_1,...,X_n \sim \text{IID N}(\mu,\sigma)$?

Consider the random variables $$X_1,X_2,...,X_n \sim \text{IID N}(\mu,\sigma^2)$$ and define the sample mean $$\bar{X} = \sum_{i=1}^n X_i/n$$.

My question: We know that $$X_i-\mu \sim \text{N}(0,\sigma^2)$$, but what is the distribution for $$X_i-\bar{X}$$?

My working so far: Since $$X_i\sim N(\mu, \sigma^2)$$ and $$\bar{X}\sim N(\mu,\frac{\sigma^2}{n})$$, if these two quantities were independent then we would have:

$$X_i-\bar{X}\sim \text{N} \bigg(0,\bigg(1+\frac{1}{n}\bigg)\sigma^2 \bigg).$$

However $$X_i$$ and $$\bar{X}$$ are not independent so I think this simple solution is not correct.

• Which part of the sample mean is independent of $X_1$? Which part not? Commented May 22, 2016 at 12:03
• If I understand right $\frac{X_2+X_3+...+X_n}{n}$ is independent of $X_1$, $\frac{X_1}{n}$ is not Commented May 22, 2016 at 12:43
• What do you know about variances of sums or differences of correlated random variables? Commented May 23, 2016 at 6:12

Updated with full solution since OP has now solved it.

For a complete solution, one needs to first show that $Y_i:= X_i - \bar{X}$ is a Gaussian random variable, whence it suffices to find its mean and variance to characterize the distribution. Knowing something about Gaussian random vectors makes this straight-forward. This solution is presented first, and then I also provide a more direct argument without use of multivariate techniques.

Multivariate solution: If $X_i\overset{iid}{\sim} N(\mu, \sigma^2)$, then $\mathbf{x}: = [X_1, \dots, X_n]' \sim N_n(1_n\mu, \sigma^2I_n)$. We also know that for any conformable matrix $A$, $A'\mathbf{x}$ is Gaussian with mean $A'1_n\mu$ and covariance matrix $\sigma^2A'A$. Consider, without loss of generality, the case $i = 1$. Then we have $$A=[1 - 1/n, -1/n,\dots, -1/n]'.$$ That is, $Y_1$ is Gaussian with mean $A'1_n \mu = 0$ and variance $$\sigma^2 A'A = \sigma^2([1-1/n]^2+(n-1)/n^2) = \sigma^2(n-1)/n.$$ This completes the first solution.

Univariate solution: We can write $Y_i = (1-1/n)X_i - \sum_{j\neq i}X_j/n$, where the first term is independent of the second because functions of independent random variables are independent. We also know that sums of independent Gaussian random variables are still Gaussian, and that multiplying a Gaussian random variable by a constant gives another Gaussian random variable. Thus, using standard rules of means and variances, $$(1 - 1/n)X_i\sim N(0, (1-1/n)^2 \sigma^2)$$ and

$$-1\sum_{j\neq i}X_j \sim N(0, (n-1)\sigma^2/n^2),$$

which implies that $$Y_i \sim N(0, \sigma^2(n-1)/n),$$

where we have used that independence implies zero covariance.

• Thanks, should it be $A=[1 - 1/n, 1/n,\dots, 1/n]'$? Commented May 22, 2016 at 13:24
• No, I don't think so. Try the case $n = 2$. $[1 - 1/n, -1/n][X_1, X_2]' = X_1 - X_1/n - X_2/n = X_1 - (X_1 + X_2)/n = X_1 - \bar{X}$
– KOE
Commented May 22, 2016 at 13:27
• I think I also find another solution without using Matrix. I will try to post later. Commented May 22, 2016 at 13:31

Student001 already gave a solution for the question and accepted. Micheal M's comments are also very helpful, so I will post another solution following Micheal's suggestion without using matrix notation.

$X_1-\bar{X}=X_1-\frac{X_1+X_2+...+X_n}{n}\\=X_1-\frac{X_1}{n}-\frac{X_2+X_3+...+X_n}{n}\\=(1-\frac{1}{n})X_1-\frac{1}{n}(X_2+X_3+...+X_n)$

Next we know that:

$(1-\frac{1}{n})X_1\sim N(\frac{n-1}{n}\mu,\frac{(n-1)^2}{n^2}\sigma^2) \tag 1$

$\frac{1}{n}(X_2+X_3+...+X_n)\sim \frac{1}{n}N((n-1)\mu,(n-1)\sigma^2)=N(\frac{n-1}{n},\frac{n-1}{n^2}\sigma^2) \tag 2$

And $(1)-(2)$ has a $N(0,\frac{(n-1)^2+n-1}{n^2}\sigma^2)=N(0,\frac{n-1}{n}\sigma^2)$

i.e $X_1-\bar{X}\sim N(0,\frac{n-1}{n}\sigma^2)$

This result is exactly the same as Student001's results.

Another method as suggest by Glen_b: we need to find the variance of $X_i$ and $\bar{X}$

$Var(X_i-\bar{X})=Var(X_i)+Var(\bar{X})-2Cov(X_i,\bar{X})$

The key is to calculate the $Cov(X_i,\bar{X})$

$Cov(X_i,\bar{X})=Cov(X_i, \frac{1}{n}(X_1+X_2+...+X_i+...+X_n))$

We will use the formula: $Cov(X,Y+Z)=Cov(X,Y)+Cov(X,Z)$

$Cov(X_i,\frac{1}{n}(X_1+X_2+...+X_i+...+X_n)=cov(X_i,\frac{1}{n}X_1)+...+Cov(X_i,\frac{1}{n}X_i)+...+Cov(X_i,\frac{1}{n}X_n)$

By i.i.d we know that except $Cov(X_i,\frac{1}{n}X_i)$ all other terms are zeros.

$\therefore Cov(X_i,\bar{X})=Cov(X_i,\frac{1}{n}X_i)=\frac{1}{n}Cov(X_i,X_i)=\frac{1}{n}Var(X_i)=\frac{1}{n}\sigma^2$

Finally,

$Var(X_i-\bar{X})=Var(X_i)+Var(\bar{X})-2Cov(X_i,\bar{X})=\sigma^2+\frac{\sigma^2}{n}-2\frac{1}{n}\sigma^2=\frac{n-1}{n}\sigma^2$

All methods get same results.

Staying in the univariate case, since $$X_i,~ i=1 ,\dots ,N$$ are iid Normally distributed with mean $$\mu$$ and variance $$\sigma^2$$, we have, as you mentioned,

$$E[X_i -\bar X] = E\left[X_i- \frac{1}{n}\sum_j^N X_j\right] = E[X_i] - \frac{1}{n}\sum_j^N E[X_j] = \mu - \frac{1}{n}n\mu =0.$$

For the variance, notice that

$$\operatorname{Var}[X_i-\bar X] = \operatorname{Var}[X_i] + \operatorname{Var}[\bar{X}] - 2\operatorname{Cov}(X_i, \bar{X})$$

Where

$$\operatorname{Cov}(X_i, \bar{X}) = \operatorname{Cov}\left(X_i, \frac{1}{n}\sum_j^N X_j\right) = \operatorname{Cov}\left(X_i, \frac{1}{n}X_i\right)$$ by independence.

Also, you technically have to check that the resulting distribution is itself normal. Then, using the shortcut argument that the normal distribution is entirely determined by its first two moments, we have that:

$$X_i - \bar X \sim N \left (E[X_i - \bar X], \operatorname{Var}[X_i - \bar X] \right ).$$

• Welcome to the site! Please note that there is a reason these calculations were left out from my answer: this is a self-study question. You may read about it here: stats.stackexchange.com/tags/self-study/info
– KOE
Commented May 22, 2016 at 14:05
• Oh I see, thank you. I edited my solution so that it does not provide the answer. However, I think that pedagogically, going to the multivariate case is overkilling it so maybe OP can still benefit from this.
– wiwh
Commented May 22, 2016 at 14:14
• I indeed missed a part. It is now orrected, thanks. Student001's answer is complete imho, especially with the justification of why we end up with a Gaussian RV.
– wiwh
Commented May 24, 2016 at 7:28

#### Here is the multivariate extension using the centring matrix

While there are other answers here that solve your problem, you might be interested to know that this problem is closely related to the properties of the centring matrix, which takes a vector of values and centres them around their sample mean. Starting with a random vector $$\mathbf{X}$$ you can get the corresponding centred vector (i.e., the vector of deviations from the sample mean) as:

$$\mathbf{Y} \equiv \mathbf{X} - \bar{X}_n \mathbf{1} = \mathbf{C} \mathbf{X},$$

where the $$n \times n$$ centring matrix $$\mathbf{C}$$ is given by:

$$\mathbf{C} \equiv \frac{1}{n} \begin{bmatrix} n-1 & -1 & -1 & \cdots & -1 \\ -1 & n-1 & -1 & \cdots & -1 \\ -1 & -1 & n-1 & \cdots & -1 \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ -1 & -1 & -1 & \cdots & n-1 \\ \end{bmatrix}.$$

(You can find some statistical theory related to the centring matrix and other matrices with a similar form in O'Neill (2021).) Now, if the values in the initial vector are IID with mean $$\mu$$ and variance $$\sigma^2$$ then we have $$\mathbb{E}(\mathbf{X}) = \mu \mathbf{1}$$ and $$\mathbb{V}(\mathbf{X}) = \sigma^2 \mathbf{I}$$. applying the standard rules for the moments of linear transformations gives:

\begin{align} \mathbb{E}(\mathbf{Y}) &= \mathbb{E}(\mathbf{C} \mathbf{X}) \\[6pt] &= \mathbf{C} \mathbb{E}(\mathbf{X}) \\[6pt] &= \mu \mathbf{C} \mathbf{1} \\[6pt] &= \mu \mathbf{0} \\[6pt] &= \mathbf{0}, \\[6pt] \mathbb{V}(\mathbf{Y}) &= \mathbb{V}(\mathbf{C} \mathbf{X}) \\[6pt] &= \mathbf{C} \mathbb{V}(\mathbf{X}) \mathbf{C}^\text{T} \\[6pt] &= \sigma^2 \mathbf{C} \mathbf{I} \mathbf{C}^\text{T} \\[6pt] &= \sigma^2 \mathbf{C}^2 \\[6pt] &= \sigma^2 \mathbf{C}. \\[6pt] \end{align}

In the case where $$\mathbf{X}$$ is (multivariate) normally distributed, this distribution is preserved under linear transformation and so we have:

$$\mathbf{Y} \sim \text{N}(\mathbf{0}, \sigma^2 \mathbf{C}).$$

This multivariate distribution implies that for the individual elements we have marginal distribution $$Y_i = X_i - \bar{X}_n \sim \text{N}(0, \tfrac{n-1}{n} \cdot \sigma^2)$$. (It is also useful to note that the individual elements $$Y_1,...,Y_n$$ are negatively correlated with one another.)