What is the distribution of $X_i-\bar{X}$ when $X_1,...,X_n \sim \text{IID N}(\mu,\sigma)$?

Question

Consider the random variables $X_1,X_2,...,X_n \sim \text{IID N}(\mu,\sigma^2)$ and define the sample mean $\bar{X} = \sum_{i=1}^n X_i/n$.

My question: We know that $X_i-\mu \sim \text{N}(0,\sigma^2)$, but what is the distribution for $X_i-\bar{X}$?

---

My working so far: Since $X_i\sim N(\mu, \sigma^2)$ and $\bar{X}\sim N(\mu,\frac{\sigma^2}{n})$, if these two quantities were independent then we would have:

$$X_i-\bar{X}\sim \text{N} \bigg(0,\bigg(1+\frac{1}{n}\bigg)\sigma^2 \bigg).$$

However $X_i$ and $\bar{X}$ are not independent so I think this simple solution is not correct.

Answer 1

Updated with full solution since OP has now solved it.

---

For a complete solution, one needs to first show that $ Y_i:= X_i - \bar{X}$ is a Gaussian random variable, whence it suffices to find its mean and variance to characterize the distribution. Knowing something about Gaussian random vectors makes this straight-forward. This solution is presented first, and then I also provide a more direct argument without use of multivariate techniques.

Multivariate solution: If $X_i\overset{iid}{\sim} N(\mu, \sigma^2)$, then $\mathbf{x}: = [X_1, \dots, X_n]' \sim N_n(1_n\mu, \sigma^2I_n)$. We also know that for any conformable matrix $A$, $A'\mathbf{x}$ is Gaussian with mean $A'1_n\mu$ and covariance matrix $\sigma^2A'A$. Consider, without loss of generality, the case $i = 1$. Then we have $$A=[1 - 1/n, -1/n,\dots, -1/n]'.$$ That is, $Y_1$ is Gaussian with mean $A'1_n \mu = 0$ and variance $$\sigma^2 A'A = \sigma^2([1-1/n]^2+(n-1)/n^2) = \sigma^2(n-1)/n.$$ This completes the first solution.

---

Univariate solution: We can write $Y_i = (1-1/n)X_i - \sum_{j\neq i}X_j/n$, where the first term is independent of the second because functions of independent random variables are independent. We also know that sums of independent Gaussian random variables are still Gaussian, and that multiplying a Gaussian random variable by a constant gives another Gaussian random variable. Thus, using standard rules of means and variances, $$(1 - 1/n)X_i\sim N(0, (1-1/n)^2 \sigma^2)$$ and

$$ -1\sum_{j\neq i}X_j \sim N(0, (n-1)\sigma^2/n^2), $$

which implies that $$Y_i \sim N(0, \sigma^2(n-1)/n),$$

where we have used that independence implies zero covariance.

Answer 2

Student001 already gave a solution for the question and accepted. Micheal M's comments are also very helpful, so I will post another solution following Micheal's suggestion without using matrix notation.

$X_1-\bar{X}=X_1-\frac{X_1+X_2+...+X_n}{n}\\=X_1-\frac{X_1}{n}-\frac{X_2+X_3+...+X_n}{n}\\=(1-\frac{1}{n})X_1-\frac{1}{n}(X_2+X_3+...+X_n)$

Next we know that:

$(1-\frac{1}{n})X_1\sim N(\frac{n-1}{n}\mu,\frac{(n-1)^2}{n^2}\sigma^2) \tag 1$

$\frac{1}{n}(X_2+X_3+...+X_n)\sim \frac{1}{n}N((n-1)\mu,(n-1)\sigma^2)=N(\frac{n-1}{n},\frac{n-1}{n^2}\sigma^2) \tag 2$

And $(1)-(2)$ has a $N(0,\frac{(n-1)^2+n-1}{n^2}\sigma^2)=N(0,\frac{n-1}{n}\sigma^2)$

i.e $X_1-\bar{X}\sim N(0,\frac{n-1}{n}\sigma^2)$

This result is exactly the same as Student001's results.

---

Another method as suggest by Glen_b: we need to find the variance of $X_i$ and $\bar{X}$

$Var(X_i-\bar{X})=Var(X_i)+Var(\bar{X})-2Cov(X_i,\bar{X})$

The key is to calculate the $Cov(X_i,\bar{X})$

$Cov(X_i,\bar{X})=Cov(X_i, \frac{1}{n}(X_1+X_2+...+X_i+...+X_n))$

We will use the formula: $Cov(X,Y+Z)=Cov(X,Y)+Cov(X,Z)$

$Cov(X_i,\frac{1}{n}(X_1+X_2+...+X_i+...+X_n)=cov(X_i,\frac{1}{n}X_1)+...+Cov(X_i,\frac{1}{n}X_i)+...+Cov(X_i,\frac{1}{n}X_n)$

By i.i.d we know that except $Cov(X_i,\frac{1}{n}X_i)$ all other terms are zeros.

$\therefore Cov(X_i,\bar{X})=Cov(X_i,\frac{1}{n}X_i)=\frac{1}{n}Cov(X_i,X_i)=\frac{1}{n}Var(X_i)=\frac{1}{n}\sigma^2$

Finally,

$Var(X_i-\bar{X})=Var(X_i)+Var(\bar{X})-2Cov(X_i,\bar{X})=\sigma^2+\frac{\sigma^2}{n}-2\frac{1}{n}\sigma^2=\frac{n-1}{n}\sigma^2$

All methods get same results.

Answer 3

Staying in the univariate case, since $X_i,~ i=1 ,\dots ,N$ are iid Normally distributed with mean $\mu$ and variance $\sigma^2$, we have, as you mentioned,

$$E[X_i -\bar X] = E\left[X_i- \frac{1}{n}\sum_j^N X_j\right] = E[X_i] - \frac{1}{n}\sum_j^N E[X_j] = \mu - \frac{1}{n}n\mu =0.$$

For the variance, notice that

$$\operatorname{Var}[X_i-\bar X] = \operatorname{Var}[X_i] + \operatorname{Var}[\bar{X}] - 2\operatorname{Cov}(X_i, \bar{X})$$

Where

$$\operatorname{Cov}(X_i, \bar{X}) = \operatorname{Cov}\left(X_i, \frac{1}{n}\sum_j^N X_j\right) = \operatorname{Cov}\left(X_i, \frac{1}{n}X_i\right)$$ by independence.

This should lead you to your answer.

Also, you technically have to check that the resulting distribution is itself normal. Then, using the shortcut argument that the normal distribution is entirely determined by its first two moments, we have that:

$$X_i - \bar X \sim N \left (E[X_i - \bar X], \operatorname{Var}[X_i - \bar X] \right ).$$

Answer 4

Here is the multivariate extension using the centring matrix

While there are other answers here that solve your problem, you might be interested to know that this problem is closely related to the properties of the centring matrix, which takes a vector of values and centres them around their sample mean. Starting with a random vector $\mathbf{X}$ you can get the corresponding centred vector (i.e., the vector of deviations from the sample mean) as:

$$\mathbf{Y} \equiv \mathbf{X} - \bar{X}_n \mathbf{1} = \mathbf{C} \mathbf{X},$$

where the $n \times n$ centring matrix $\mathbf{C}$ is given by:

$$\mathbf{C} \equiv \frac{1}{n} \begin{bmatrix} n-1 & -1 & -1 & \cdots & -1 \\ -1 & n-1 & -1 & \cdots & -1 \\ -1 & -1 & n-1 & \cdots & -1 \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ -1 & -1 & -1 & \cdots & n-1 \\ \end{bmatrix}.$$

(You can find some statistical theory related to the centring matrix and other matrices with a similar form in O'Neill (2021).) Now, if the values in the initial vector are IID with mean $\mu$ and variance $\sigma^2$ then we have $\mathbb{E}(\mathbf{X}) = \mu \mathbf{1}$ and $\mathbb{V}(\mathbf{X}) = \sigma^2 \mathbf{I}$. applying the standard rules for the moments of linear transformations gives:

$$\begin{align} \mathbb{E}(\mathbf{Y}) &= \mathbb{E}(\mathbf{C} \mathbf{X}) \\[6pt] &= \mathbf{C} \mathbb{E}(\mathbf{X}) \\[6pt] &= \mu \mathbf{C} \mathbf{1} \\[6pt] &= \mu \mathbf{0} \\[6pt] &= \mathbf{0}, \\[6pt] \mathbb{V}(\mathbf{Y}) &= \mathbb{V}(\mathbf{C} \mathbf{X}) \\[6pt] &= \mathbf{C} \mathbb{V}(\mathbf{X}) \mathbf{C}^\text{T} \\[6pt] &= \sigma^2 \mathbf{C} \mathbf{I} \mathbf{C}^\text{T} \\[6pt] &= \sigma^2 \mathbf{C}^2 \\[6pt] &= \sigma^2 \mathbf{C}. \\[6pt] \end{align}$$

In the case where $\mathbf{X}$ is (multivariate) normally distributed, this distribution is preserved under linear transformation and so we have:

$$\mathbf{Y} \sim \text{N}(\mathbf{0}, \sigma^2 \mathbf{C}).$$

This multivariate distribution implies that for the individual elements we have marginal distribution $Y_i = X_i - \bar{X}_n \sim \text{N}(0, \tfrac{n-1}{n} \cdot \sigma^2)$. (It is also useful to note that the individual elements $Y_1,...,Y_n$ are negatively correlated with one another.)