2
$\begingroup$

Linear combination $C=\sum_{i=1}^{n} a_i \bar{X}_i$ is called a (estimated) contrast if $\sum_{i=1}^{n} a_i=0$. Two contrasts are called orthogonal if $\sum_{i=1}^{n} a_i b_i = 0$; simplest example would be $C_1=\bar{X}_1 - \bar{X}_2$ and $C_2=\bar{X}_1 + \bar{X}_2 - 2\bar{X}_3$. I have read the claim

Two orthogonal contrasts are statistically independent

in Statistical Design and Analysis of Experiments by Robert L. Mason (2003, 2nd edition, Page 248) . I have searched the web for a proof with no success. I have tried to prove $$f\left( \sum_{i=1}^{n} a_i \bar{X}_i = v_2 \mid \sum_{j=1}^{n} b_j \bar{X}_j = v_1\right) = f\left(\sum_{i=1}^{n} a_i \bar{X_i} = v_2\right) \tag 1 $$ or even uncorrelatedness (a weaker property) $$\mathbb{E}\left [\left(\sum_{i=1}^{n} a_i \bar{X}_i\right)\left(\sum_{j=1}^{n} b_j \bar{X}_j\right)\right ]=\mathbb{E}\left[\sum_{i=1}^{n} a_i \bar{X}_i\right]\mathbb{E}\left[\sum_{j=1}^{n} b_j \bar{X}_j\right] \tag 2$$ again with no success. It seems there are more intricacies involved which I am missing.

Edit:

I have managed to prove the uncorrelatedness $(2)$ using kjetil b halvorsen's advice. For now, let's assume there is only one factor, then $Y_{il} = \mu_{i} + \epsilon_{il}; \epsilon_{il} \overset{\text{iid}}{\sim} \mathcal{N}(0, \sigma^2)$ is the assumed model for i-th factor-level and l-th repeat test. Now let $\bar{X}_i \triangleq \frac{1}{r}\sum_{l=1}^r Y_{il}$, where $r$ is the number of repeat tests in a balanced experiment. Therefore, $$\begin{align*} &\mathbb{E}\left [\left(\sum_{i=1}^{n} a_i \bar{X}_i\right)\left(\sum_{j=1}^{n} b_j \bar{X}_j\right)\right ]=\sum_{i,j=1}^n a_ib_j \mathbb{E}\left [\bar{X}_i\bar{X}_j \right] \\ &\overset{\bar{X}_i \perp \bar{X}_j \text{ for } i \neq j}{=} \sum_{i\neq j} a_ib_j \mu_i \mu_j + \sum_{i=1}^n a_ib_i \mathbb{E}\left [\bar{X}_i^2 \right] \\ \end{align*}$$ By using $$\begin{align*} \mathbb{E}\left [\bar{X}_i^2 \right] &= \mathbb{E}\left [\left(\frac{1}{r}\sum_{l=1}^r Y_{il}\right) \left(\frac{1}{r}\sum_{l'=1}^r Y_{il'}\right)\right] = \frac{1}{r^2}\sum_{l,l'=1}^r\mathbb{E}\left [Y_{il} Y_{il'}\right] \\ &\overset{Y_{il} \perp Y_{il'} \text{ for } l \neq l'}{=} \frac{1}{r^2}\left(\sum_{l\neq l'} \mathbb{E}[Y_{il}]\mathbb{E}[Y_{il'}] + \sum_{l=1}^r \mathbb{E}\left [Y_{il}^2 \right]\right) \\ &= \frac{1}{r^2}\left( \sum_{l\neq l'} \mu_i^2 + \sum_{l=1}^r \mathbb{E}\left [(\mu_i + \epsilon_{il})^2 \right] \right) \\ &= \frac{1}{r^2}\left( r(r-1)\mu_i^2 + \sum_{l=1}^r \left (\mu_i ^ 2 + \sigma ^2 \right) \right) = \mu_i^2 + \frac{1}{r} \sigma^2 \end{align*}$$ We arrive at $$\begin{align*} &\mathbb{E}\left [\left(\sum_{i=1}^{n} a_i \bar{X}_i\right)\left(\sum_{j=1}^{n} b_j \bar{X}_j\right)\right ]= \sum_{i,j=1}^n a_ib_j \mu_i \mu_j + \frac{\sigma^2}{r}\sum_{i=1}^n a_ib_i \\ &\overset{\sum_{i=1}^n a_ib_i=0}{=} \sum_{i,j=1}^n a_ib_j \mathbb{E}[\bar{X}_i] \mathbb{E}[\bar{X}_j] = \mathbb{E}\left[\sum_{i=1}^{n} a_i \bar{X}_i\right]\mathbb{E}\left[\sum_{j=1}^{n} b_j \bar{X}_j\right] \end{align*}$$ Extending this proof to arbitrary $n$ factors and averages is straightforward; it can be messy though. For example, for a 3-factor experiment, any average $\bar{X}_{i'}$ can be expressed as $\bar{X}_{i'} \triangleq \sum_{ijkl} a_{ijk}Y_{ijkl}$, which can be replaced in the proof.

There is only one question left. Is "statistically independent" is used only loosely to mean "uncorrelated"? Or $(1)$ and $(2)$ can be proved to be equivalent in this context?

$\endgroup$
1
  • 1
    $\begingroup$ This needs the normal assumption on the errors, and using that, showing correlation is zero is enough. also, you need to define your notation $\bar{X}_i$. Are these averages based on equal number of observations, say? $\endgroup$ Commented Jun 15, 2021 at 15:20

2 Answers 2

3
$\begingroup$

The proof of uncorrelatedness can be done in a more straightforward fashion: Assume that $\bar X_i, i=1,\dots,n$ are uncorrelated. Let $$ C_1 = \sum_i a_i \bar X_i, \quad C_2 = \sum_i b_i \bar X_i, \quad $$ We have \begin{align*} \text{cov}(C_1, C_2) &= \text{cov}(\sum_i a_i \bar X_i, \sum_j b_j X_j) \\ &= \sum_{i} \sum_j a_i b_j \text{cov}(\bar X_i, \bar X_j) \\ &= \sum_{i} a_i b_i \text{var}(\bar X_i). \end{align*} The first equation is by the bilinrarity of the covariance operator. The second equality is by the cross-terms vanishing due to uncorrelatedness. Hence, $C_1$ and $C_2$ are uncorrelated if and only if $ \sum_{i} a_i b_i \text{var}(\bar X_i) = 0$.

Regarding the second question: If $\bar X_i, i = 1,\dots,n$ are (jointly) Gaussian, then uncorrelatedness is equivalent to independence. This is a well-known property of multivariate Gaussian distribution. Otherwise, in general, independence does not follow from uncorrelatedness:

For example, assume that $\bar X_1$ and $\bar X_2$ are independent and both uniformly distributed in $[-1,1]$. Consider $C_1 = \bar X_1 + \bar X_2$ and $C_2 = \bar X_1 - \bar X_2$. Then $C_1$ and $C_2$ will be uncorrelated but dependent. To see why they are dependent, you can plot the support of the density of $(C_1,C_2)$ in the 2-D plan which would be a rotated square (a diamond). Alternatively, note that if $C_1 = 2$, then we should have $\bar X_1 = \bar X_2 =1$, hence $C_2 = 0$. That is, the distribution of $C_2$ conditioned on $C_1 = 2$ is a point mass at $0$. On the other hand, argue that the distribution of $C_2$ conditioned on $C_1 = 0$ will be uniform is $[-2,2]$ (the picture helps here). This shows that they are dependent.

$\endgroup$
2
  • $\begingroup$ Very nice! Thanks! $\endgroup$
    – Esmailian
    Commented Jun 16, 2021 at 5:46
  • 1
    $\begingroup$ I added some comments about your second question. $\endgroup$
    – passerby51
    Commented Jun 16, 2021 at 13:22
0
$\begingroup$

I managed to prove the statement thanks to passerby51 for hinting the jointly Gaussian distribution. The proof goes as follows:

  1. Show averages in each contrast are independent
  2. Show contrasts are uncorrelated
  3. Use Theorem 1: nonsingular transformation of independent random variables $X_1,\dots,X_n$, where $X_i \sim \mathcal{N}(0, \sigma_i^2)$, is jointly Gaussian
  4. Use Theorem 2: jointly Gaussian random variables that are uncorrelated are independent

I am assuming (a) a fixed-effect ANOVA model $Y_{\alpha l} = \mu_{\alpha} + \epsilon_{\alpha l}; \epsilon_{\alpha l} \overset{\text{iid}}{\sim} \mathcal{N}(0, \sigma^2)$ where $\alpha$ denotes factor combination and $l$ denotes repeat test, and (b) experiment is balanced ($r$ repeat tests per combination). Accordingly, an average can be expressed as $\bar{X}_i \triangleq \frac{1}{r|I|}\sum_{\alpha \in I, l}Y_{f(i,\alpha) l}$. For example, in a 3-factor experiment, an average would be $\bar{X}_j \triangleq \frac{1}{acr}\sum_{ikl} Y_{ijkl}$ where $\alpha=ik$, $f(j,\alpha)=ijk$, and $I=\left\{(i, k) \mid i \in [1, a], k \in [1, c] \right\}$.

  1. For any two averages to be independent, I assume they do not share any $\epsilon_{\alpha l}$, i.e. each average $i$ contains an exclusive set of factor-level combinations $f(i, I)$. For example, in a 2-factor balanced experiment, for $\bar{X}_i \triangleq \frac{1}{br}\sum_{jl} Y_{ijl}$ (average response for factor-level $i$), any $\bar{X}_i$ and $\bar{X}_j$ do not share a $\epsilon_{\alpha l}$ with the same index, thus $\bar{X}_i \perp \bar{X}_j$ for $i \neq j$.

  2. As passerby51 neatly showed, two contrasts are uncorrelated iff $\sum_i a_ib_i\text{Var}(\bar{X}_i) = 0$. Now by using representation $\bar{X}_i = \frac{1}{r|I|}\sum_{\alpha \in I, l}Y_{f(i,\alpha) l}$ for averages, we have $$\text{Var}(\bar{X}_i) = \frac{1}{r^2|I|^2}\sum_{\alpha \in I,l}\sigma^2=\frac{\sigma^2}{r|I|} \Rightarrow \sum_i a_ib_i\text{Var}(\bar{X}_i) = \frac{\sigma^2}{r|I|}\sum_i a_ib_i \overset{\sum_{i=1}^n a_ib_i=0}{=} 0$$

  3. We express contrasts $C_1 = \sum_{i=1}^{n} a_i \bar{X}_i$ and $C_2 = \sum_{j=1}^{n} b_j \bar{X}_j$ in a matrix form as follows $$\begin{align*} \begin{bmatrix} C_1\\ C_2\\ \dots \end{bmatrix} &= \begin{bmatrix} a_1& \dots & a_n\\ b_1& \dots & b_n\\ . & \dots & . \end{bmatrix} \left( \begin{bmatrix} \bar{X}_1 - \mathbb{E}[\bar{X}_1] \\ \bar{X}_2 - \mathbb{E}[\bar{X}_2]\\ \dots \end{bmatrix} + \begin{bmatrix} \mathbb{E}[\bar{X}_1] \\ \mathbb{E}[\bar{X}_2] \\ \dots \end{bmatrix} \right) \\ &\Rightarrow \mathbf{C} = A\mathbf{X} + A\mathbf{\mu} = A\mathbf{X} +\mathbf{\mu'} \end{align*}$$ where $\mathbb{E}[\bar{X}_i]=\frac{1}{|I|}\sum_{\alpha \in I}\mu_{f(i,\alpha)}$, and $X_i \triangleq \bar{X}_i - \mathbb{E}[\bar{X}_i] = \frac{1}{r|I|}\sum_{\alpha \in I, l}\epsilon_{f(i,\alpha) l} \overset{\text{iid}}{\sim} \mathcal{N}(0, \sigma^2/r|I|)$. The other rows of matrix $A$ (3rd row and below) are filled with dummy values just to have $n$ linearly independent rows, thus a nonsingular $A$; here orthogonality of contrasts came into play. Now using the fact that jointly Gaussian distribution is closed under marginalization of the rest of (dummy) contrasts, Theorem 1 implies that $C_1$ and $C_2$ are jointly Gaussian.

  4. Using Theorem 2, statistical independence of two orthogonal contrasts follows immediately.

Theorems 1 and 2 can be found in this Lecture by Prof. Robert B. Ash

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.