# Independence under linear transformations

What is the (largest) set of matrices $$\mathcal C\subset \mathbb R^{m\times n}$$ ($$m\le n$$) for which the following statement is true?

Let $$x_1,\ldots,x_n\in\mathbb R$$ be independent random variables. Then also $$z_1, \ldots, z_m$$ are independent, given $$z=Ax$$ with $$A\in\mathcal C$$

It obviously holds when $$A$$ is a (rectangular) diagonal matrix. Is it also true if the rows are (1) orthogonal (2) linearly independent?

Conjecture: It is only true for matrices for which it is trivially true, i.e. when there is no interaction between any of the $$x_i$$: $$\mathcal C = \{DP\mid D\in\mathbb R^{m\times n} \text{ diagonal, } P\in\mathbb R^{n\times n} \text{ permutation matrix} \}$$

• linear independent a definite no, think about $$A = \begin{bmatrix} 1 & 1\\ 0 & 1 \end{bmatrix}$$ $Ax = [X,X+Y]$ where $X,X+Y$ are definitely not independent. – David Veitch Feb 20 at 14:01

This is the set $$C_{m,n}$$ of matrices that have at most one nonzero entry in each column.

Let us rephrase the issue in the question as a property of an $$m\times n$$ matrix $$A:$$

$$\mathcal{P}(A):$$ for all independent random variables $$x=(x_1,\ldots,x_n)$$ defined on a given space $$\Omega,$$ $$z=Ax$$ is independent.

My assertion requires two demonstrations: namely, that $$A\in C_{m,n}$$ implies $$\mathcal{P}(A)$$ and then that $$A\notin C_{m,n}$$ implies $$\mathcal{P}(A)$$ is false.

First, when $$A\in C_{m,n}$$ and the $$x_j$$ are independent, we have to show the $$z_i$$ are independent. This can be done with an induction on $$m.$$ When $$m=1,$$ $$\{z_1\}$$ is trivially independent. Now assume the result for $$m-1.$$ The structure of $$A$$ is such that $$z_m$$ can be expressed in terms of the $$x_j$$ for which $$A_{mj}\ne 0$$ and all the other $$z_i,$$ $$i\lt m,$$ can be expressed in terms of the remaining $$x_j.$$ Because the $$x_j$$ are independent, $$z_m$$ is therefore independent of the remaining $$z_i,$$ whence $$\{z_1,\ldots,z_m\}$$ is independent. That establishes the first claim.

Second, suppose $$A \notin C_{m,n}.$$ This means there is a column $$j$$ and two rows $$i,i^\prime$$ for which $$A_{ij} \ne 0$$ and $$A_{i^\prime j}\ne 0.$$ We may therefore write

$$z_i = A_{ij} x_j + y_i,\quad z_{i^\prime} = A_{i^\prime j} x_j + y_{i^\prime}$$

for variables $$y_i$$ and $$y_{i^\prime}$$ formed out of the remaining $$x_{j^\prime},$$ $$j^\prime\ne j.$$ Setting all such $$x_{j^\prime}$$ to be constant and selecting $$x_j$$ to have unit variance, compute

$$\operatorname{Cov}(z_i,z_{i^\prime}) = A_{ij}A_{i^\prime j} \ne 0,$$

showing $$z_i$$ and $$z_{i^\prime}$$ are not independent, which means $$\mathcal{P}(A)$$ is false, QED.

For orthogonal columns, in general it's not. Let $$A=\begin{bmatrix}1 &-1\\1&1\end{bmatrix}$$

This results in $$z_1=x_1-x_2,z_2=x_1+x_2$$. These two aren't independent in general. For example, assume $$x_i$$ are Bernoulli RVs, then if $$z_2=2$$, $$z_1$$ is definitely $$0$$, which means there is dependency.

For the linear independence, see @David's counter-example.