# Prove two orthogonal contrasts are statistically independent

Linear combination $$C=\sum_{i=1}^{n} a_i \bar{X}_i$$ is called a (estimated) contrast if $$\sum_{i=1}^{n} a_i=0$$. Two contrasts are called orthogonal if $$\sum_{i=1}^{n} a_i b_i = 0$$; simplest example would be $$C_1=\bar{X}_1 - \bar{X}_2$$ and $$C_2=\bar{X}_1 + \bar{X}_2 - 2\bar{X}_3$$. I have read the claim

Two orthogonal contrasts are statistically independent

in Statistical Design and Analysis of Experiments by Robert L. Mason (2003, 2nd edition, Page 248) . I have searched the web for a proof with no success. I have tried to prove $$f\left( \sum_{i=1}^{n} a_i \bar{X}_i = v_2 \mid \sum_{j=1}^{n} b_j \bar{X}_j = v_1\right) = f\left(\sum_{i=1}^{n} a_i \bar{X_i} = v_2\right) \tag 1$$ or even uncorrelatedness (a weaker property) $$\mathbb{E}\left [\left(\sum_{i=1}^{n} a_i \bar{X}_i\right)\left(\sum_{j=1}^{n} b_j \bar{X}_j\right)\right ]=\mathbb{E}\left[\sum_{i=1}^{n} a_i \bar{X}_i\right]\mathbb{E}\left[\sum_{j=1}^{n} b_j \bar{X}_j\right] \tag 2$$ again with no success. It seems there are more intricacies involved which I am missing.

### Edit:

I have managed to prove the uncorrelatedness $$(2)$$ using kjetil b halvorsen's advice. For now, let's assume there is only one factor, then $$Y_{il} = \mu_{i} + \epsilon_{il}; \epsilon_{il} \overset{\text{iid}}{\sim} \mathcal{N}(0, \sigma^2)$$ is the assumed model for i-th factor-level and l-th repeat test. Now let $$\bar{X}_i \triangleq \frac{1}{r}\sum_{l=1}^r Y_{il}$$, where $$r$$ is the number of repeat tests in a balanced experiment. Therefore, \begin{align*} &\mathbb{E}\left [\left(\sum_{i=1}^{n} a_i \bar{X}_i\right)\left(\sum_{j=1}^{n} b_j \bar{X}_j\right)\right ]=\sum_{i,j=1}^n a_ib_j \mathbb{E}\left [\bar{X}_i\bar{X}_j \right] \\ &\overset{\bar{X}_i \perp \bar{X}_j \text{ for } i \neq j}{=} \sum_{i\neq j} a_ib_j \mu_i \mu_j + \sum_{i=1}^n a_ib_i \mathbb{E}\left [\bar{X}_i^2 \right] \\ \end{align*} By using \begin{align*} \mathbb{E}\left [\bar{X}_i^2 \right] &= \mathbb{E}\left [\left(\frac{1}{r}\sum_{l=1}^r Y_{il}\right) \left(\frac{1}{r}\sum_{l'=1}^r Y_{il'}\right)\right] = \frac{1}{r^2}\sum_{l,l'=1}^r\mathbb{E}\left [Y_{il} Y_{il'}\right] \\ &\overset{Y_{il} \perp Y_{il'} \text{ for } l \neq l'}{=} \frac{1}{r^2}\left(\sum_{l\neq l'} \mathbb{E}[Y_{il}]\mathbb{E}[Y_{il'}] + \sum_{l=1}^r \mathbb{E}\left [Y_{il}^2 \right]\right) \\ &= \frac{1}{r^2}\left( \sum_{l\neq l'} \mu_i^2 + \sum_{l=1}^r \mathbb{E}\left [(\mu_i + \epsilon_{il})^2 \right] \right) \\ &= \frac{1}{r^2}\left( r(r-1)\mu_i^2 + \sum_{l=1}^r \left (\mu_i ^ 2 + \sigma ^2 \right) \right) = \mu_i^2 + \frac{1}{r} \sigma^2 \end{align*} We arrive at \begin{align*} &\mathbb{E}\left [\left(\sum_{i=1}^{n} a_i \bar{X}_i\right)\left(\sum_{j=1}^{n} b_j \bar{X}_j\right)\right ]= \sum_{i,j=1}^n a_ib_j \mu_i \mu_j + \frac{\sigma^2}{r}\sum_{i=1}^n a_ib_i \\ &\overset{\sum_{i=1}^n a_ib_i=0}{=} \sum_{i,j=1}^n a_ib_j \mathbb{E}[\bar{X}_i] \mathbb{E}[\bar{X}_j] = \mathbb{E}\left[\sum_{i=1}^{n} a_i \bar{X}_i\right]\mathbb{E}\left[\sum_{j=1}^{n} b_j \bar{X}_j\right] \end{align*} Extending this proof to arbitrary $$n$$ factors and averages is straightforward; it can be messy though. For example, for a 3-factor experiment, any average $$\bar{X}_{i'}$$ can be expressed as $$\bar{X}_{i'} \triangleq \sum_{ijkl} a_{ijk}Y_{ijkl}$$, which can be replaced in the proof.

There is only one question left. Is "statistically independent" is used only loosely to mean "uncorrelated"? Or $$(1)$$ and $$(2)$$ can be proved to be equivalent in this context?

• This needs the normal assumption on the errors, and using that, showing correlation is zero is enough. also, you need to define your notation $\bar{X}_i$. Are these averages based on equal number of observations, say? Jun 15, 2021 at 15:20

The proof of uncorrelatedness can be done in a more straightforward fashion: Assume that $$\bar X_i, i=1,\dots,n$$ are uncorrelated. Let $$C_1 = \sum_i a_i \bar X_i, \quad C_2 = \sum_i b_i \bar X_i, \quad$$ We have \begin{align*} \text{cov}(C_1, C_2) &= \text{cov}(\sum_i a_i \bar X_i, \sum_j b_j X_j) \\ &= \sum_{i} \sum_j a_i b_j \text{cov}(\bar X_i, \bar X_j) \\ &= \sum_{i} a_i b_i \text{var}(\bar X_i). \end{align*} The first equation is by the bilinrarity of the covariance operator. The second equality is by the cross-terms vanishing due to uncorrelatedness. Hence, $$C_1$$ and $$C_2$$ are uncorrelated if and only if $$\sum_{i} a_i b_i \text{var}(\bar X_i) = 0$$.

Regarding the second question: If $$\bar X_i, i = 1,\dots,n$$ are (jointly) Gaussian, then uncorrelatedness is equivalent to independence. This is a well-known property of multivariate Gaussian distribution. Otherwise, in general, independence does not follow from uncorrelatedness:

For example, assume that $$\bar X_1$$ and $$\bar X_2$$ are independent and both uniformly distributed in $$[-1,1]$$. Consider $$C_1 = \bar X_1 + \bar X_2$$ and $$C_2 = \bar X_1 - \bar X_2$$. Then $$C_1$$ and $$C_2$$ will be uncorrelated but dependent. To see why they are dependent, you can plot the support of the density of $$(C_1,C_2)$$ in the 2-D plan which would be a rotated square (a diamond). Alternatively, note that if $$C_1 = 2$$, then we should have $$\bar X_1 = \bar X_2 =1$$, hence $$C_2 = 0$$. That is, the distribution of $$C_2$$ conditioned on $$C_1 = 2$$ is a point mass at $$0$$. On the other hand, argue that the distribution of $$C_2$$ conditioned on $$C_1 = 0$$ will be uniform is $$[-2,2]$$ (the picture helps here). This shows that they are dependent.

• Very nice! Thanks! Jun 16, 2021 at 5:46

I managed to prove the statement thanks to passerby51 for hinting the jointly Gaussian distribution. The proof goes as follows:

1. Show averages in each contrast are independent
2. Show contrasts are uncorrelated
3. Use Theorem 1: nonsingular transformation of independent random variables $$X_1,\dots,X_n$$, where $$X_i \sim \mathcal{N}(0, \sigma_i^2)$$, is jointly Gaussian
4. Use Theorem 2: jointly Gaussian random variables that are uncorrelated are independent

I am assuming (a) a fixed-effect ANOVA model $$Y_{\alpha l} = \mu_{\alpha} + \epsilon_{\alpha l}; \epsilon_{\alpha l} \overset{\text{iid}}{\sim} \mathcal{N}(0, \sigma^2)$$ where $$\alpha$$ denotes factor combination and $$l$$ denotes repeat test, and (b) experiment is balanced ($$r$$ repeat tests per combination). Accordingly, an average can be expressed as $$\bar{X}_i \triangleq \frac{1}{r|I|}\sum_{\alpha \in I, l}Y_{f(i,\alpha) l}$$. For example, in a 3-factor experiment, an average would be $$\bar{X}_j \triangleq \frac{1}{acr}\sum_{ikl} Y_{ijkl}$$ where $$\alpha=ik$$, $$f(j,\alpha)=ijk$$, and $$I=\left\{(i, k) \mid i \in [1, a], k \in [1, c] \right\}$$.

1. For any two averages to be independent, I assume they do not share any $$\epsilon_{\alpha l}$$, i.e. each average $$i$$ contains an exclusive set of factor-level combinations $$f(i, I)$$. For example, in a 2-factor balanced experiment, for $$\bar{X}_i \triangleq \frac{1}{br}\sum_{jl} Y_{ijl}$$ (average response for factor-level $$i$$), any $$\bar{X}_i$$ and $$\bar{X}_j$$ do not share a $$\epsilon_{\alpha l}$$ with the same index, thus $$\bar{X}_i \perp \bar{X}_j$$ for $$i \neq j$$.

2. As passerby51 neatly showed, two contrasts are uncorrelated iff $$\sum_i a_ib_i\text{Var}(\bar{X}_i) = 0$$. Now by using representation $$\bar{X}_i = \frac{1}{r|I|}\sum_{\alpha \in I, l}Y_{f(i,\alpha) l}$$ for averages, we have $$\text{Var}(\bar{X}_i) = \frac{1}{r^2|I|^2}\sum_{\alpha \in I,l}\sigma^2=\frac{\sigma^2}{r|I|} \Rightarrow \sum_i a_ib_i\text{Var}(\bar{X}_i) = \frac{\sigma^2}{r|I|}\sum_i a_ib_i \overset{\sum_{i=1}^n a_ib_i=0}{=} 0$$

3. We express contrasts $$C_1 = \sum_{i=1}^{n} a_i \bar{X}_i$$ and $$C_2 = \sum_{j=1}^{n} b_j \bar{X}_j$$ in a matrix form as follows \begin{align*} \begin{bmatrix} C_1\\ C_2\\ \dots \end{bmatrix} &= \begin{bmatrix} a_1& \dots & a_n\\ b_1& \dots & b_n\\ . & \dots & . \end{bmatrix} \left( \begin{bmatrix} \bar{X}_1 - \mathbb{E}[\bar{X}_1] \\ \bar{X}_2 - \mathbb{E}[\bar{X}_2]\\ \dots \end{bmatrix} + \begin{bmatrix} \mathbb{E}[\bar{X}_1] \\ \mathbb{E}[\bar{X}_2] \\ \dots \end{bmatrix} \right) \\ &\Rightarrow \mathbf{C} = A\mathbf{X} + A\mathbf{\mu} = A\mathbf{X} +\mathbf{\mu'} \end{align*} where $$\mathbb{E}[\bar{X}_i]=\frac{1}{|I|}\sum_{\alpha \in I}\mu_{f(i,\alpha)}$$, and $$X_i \triangleq \bar{X}_i - \mathbb{E}[\bar{X}_i] = \frac{1}{r|I|}\sum_{\alpha \in I, l}\epsilon_{f(i,\alpha) l} \overset{\text{iid}}{\sim} \mathcal{N}(0, \sigma^2/r|I|)$$. The other rows of matrix $$A$$ (3rd row and below) are filled with dummy values just to have $$n$$ linearly independent rows, thus a nonsingular $$A$$; here orthogonality of contrasts came into play. Now using the fact that jointly Gaussian distribution is closed under marginalization of the rest of (dummy) contrasts, Theorem 1 implies that $$C_1$$ and $$C_2$$ are jointly Gaussian.

4. Using Theorem 2, statistical independence of two orthogonal contrasts follows immediately.

Theorems 1 and 2 can be found in this Lecture by Prof. Robert B. Ash