Skip to main content
added 254 characters in body
Source Link
Sextus Empiricus
  • 86.5k
  • 6
  • 115
  • 302

###Linear independence is not just sufficient but also a neccesary condition

To show that the variance-covariance matrix has eigenvalues equal to zero if and only if the variables are not linearly independent, it only remains to be shown that "if the matrix has eigenvalues equal to zero then the variables are not linearly independent".

If you have a zero eigenvalue for $C_{ij} = \text{Cov}(X_i,X_j)$ then there is some linear combination (defined by the eigenvector $v$)

$$Y = \sum_{i=1}^n v_i (X_i) $$

such that

$$\begin{array}{rcl} \text{Cov}(Y,Y) &=& \sum_{i=1}^n \sum_{j=1}^n v_i v_j \text{Cov}(X_i,X_j) \\ &=&\sum_{i=1}^n v_i\sum_{j=1}^n v_j C_{ij} \\ &= &\sum_{i=1}^n v_i \cdot 0 \\ &=& 0 \end{array}$$

which means that $Y$ needs to be a constant and thus the variables $X_i$ have to add up to a constant and are either constants themselves (the trivial case) or not linearly independent.


 

###Notes about derivation - the first line in the equation with $\text{Cov}(Y,Y)$ is due to the property of covariance $$\scriptsize\text{Cov}(aU+bV,cW+dX) = ac\,\text{Cov}(U,W) + bc\,\text{Cov}(V,W) +ad\, \text{Cov}(U,X) + bd \,\text{Cov}(V,X) $$

  • the first line in the equation with $\text{Cov}(Y,Y)$ is due to the property of covariance $$\scriptsize\text{Cov}(aU+bV,cW+dX) = ac\,\text{Cov}(U,W) + bc\,\text{Cov}(V,W) +ad\, \text{Cov}(U,X) + bd \,\text{Cov}(V,X) $$

  • the step from the second to the third line is due to the property of a zero eigenvalue $$\scriptsize \sum_{j=1}^nv_jC_{ij} = 0$$

- the step from the second to the third line is due to the property of a zero eigenvalue $$\scriptsize \sum_{j=1}^nv_jC_{ij} = 0$$


Non-linear constraints

So, since linear constraints are a necessary condition (not just sufficient), non-linear constraints will only be relevant when they indirectly imply a (necessary) linear constraint.

In fact, there is a direct correspondence between the eigenvectors associated with the zero eigenvalue and the linear constraints.

$$C \cdot v = 0 \iff Y = \sum_{i=1}^n v_i X_i = \text{const}$$

Thus non-linear constraints leading to a zero eigenvalue must, together combined, generate some linear constraint.


How can non-linear constraints lead to linear constraints

Your example in the comments is an example where the intersections ofcan show this intuitively how non-linear constraints become effectively acan lead to linear constraintconstraints by reversing the derivation. The following non-linear constraints

With the matrix $$M=\begin{bmatrix} X_1 & X_2 \\ X_3 & X_4\end{bmatrix}$$ the constraint $M \cdot M^T = I$$$\begin{array}{lcr} a^2+b^2&=&1\\ c^2+d^2&=&1\\ ac + bd &=& 0 \\ ad - bc &=& 1 \end{array}$$

can be translatedreduced to the variables parameterized as:

$$\begin{array}{rcl} X_1 &=& cos(\alpha) \\ X_2 &=& sin(\alpha) \\ X_3 &=& sin(\alpha +(1+k) \pi) \\ X_4 &=& cos(\alpha +k \pi) \\\end{array}$$$$\begin{array}{lcr} a^2+b^2&=&1\\ c^2+d^2&=&1\\ a-d&=&0 \\ b+c &=& 0 \end{array}$$

You could inverse this. Say you have non-linear plus linear constraints, then it is not strange to imagine how we can replace one of the linear constraints with a fourthnon-linear constraint, likeby filling the linear constraints into the non-linear constraints. E.g when we substitute $\det M=1$, this reduces further to:

$$\begin{array}{rcrcr} X_1 &=& cos(\alpha) \\ X_2 &=& sin(\alpha) \\ X_3 &=& -sin(\alpha) &=& -X_2 \\ X_4 &=& cos(\alpha) &=& X_1 \\\end{array}$$$a=d$ and $b=-c$ in the non-linear form $a^2+b^2=1$ then you can make another relationship $ad-bc=1$. And when you multiply $a=d$ and $c=-b$ then you get $ac=-bd$.

###Linear independence is not just sufficient but also a neccesary condition

To show that the variance-covariance matrix has eigenvalues equal to zero if and only if the variables are not linearly independent, it only remains to be shown that "if the matrix has eigenvalues equal to zero then the variables are not linearly independent".

If you have a zero eigenvalue for $C_{ij} = \text{Cov}(X_i,X_j)$ then there is some linear combination (defined by the eigenvector $v$)

$$Y = \sum_{i=1}^n v_i (X_i) $$

such that

$$\begin{array}{rcl} \text{Cov}(Y,Y) &=& \sum_{i=1}^n \sum_{j=1}^n v_i v_j \text{Cov}(X_i,X_j) \\ &=&\sum_{i=1}^n v_i\sum_{j=1}^n v_j C_{ij} \\ &= &\sum_{i=1}^n v_i \cdot 0 \\ &=& 0 \end{array}$$

which means that $Y$ needs to be a constant and thus the variables $X_i$ have to add up to a constant and are either constants themselves (the trivial case) or not linearly independent.


 

###Notes about derivation

  • the first line in the equation with $\text{Cov}(Y,Y)$ is due to the property of covariance $$\scriptsize\text{Cov}(aU+bV,cW+dX) = ac\,\text{Cov}(U,W) + bc\,\text{Cov}(V,W) +ad\, \text{Cov}(U,X) + bd \,\text{Cov}(V,X) $$

  • the step from the second to the third line is due to the property of a zero eigenvalue $$\scriptsize \sum_{j=1}^nv_jC_{ij} = 0$$


Non-linear constraints

So, since linear constraints are a necessary condition (not just sufficient), non-linear constraints will only be relevant when they indirectly imply a (necessary) linear constraint.

In fact, there is a direct correspondence between the eigenvectors associated with the zero eigenvalue and the linear constraints.

$$C \cdot v = 0 \iff Y = \sum_{i=1}^n v_i X_i = \text{const}$$

Thus non-linear constraints leading to a zero eigenvalue must, together combined, generate some linear constraint.

Your example in the comments is an example where the intersections of non-linear constraints become effectively a linear constraint.

With the matrix $$M=\begin{bmatrix} X_1 & X_2 \\ X_3 & X_4\end{bmatrix}$$ the constraint $M \cdot M^T = I$ can be translated to the variables parameterized as:

$$\begin{array}{rcl} X_1 &=& cos(\alpha) \\ X_2 &=& sin(\alpha) \\ X_3 &=& sin(\alpha +(1+k) \pi) \\ X_4 &=& cos(\alpha +k \pi) \\\end{array}$$

a fourth constraint, like $\det M=1$, this reduces further to:

$$\begin{array}{rcrcr} X_1 &=& cos(\alpha) \\ X_2 &=& sin(\alpha) \\ X_3 &=& -sin(\alpha) &=& -X_2 \\ X_4 &=& cos(\alpha) &=& X_1 \\\end{array}$$

###Linear independence is not just sufficient but also a neccesary condition

To show that the variance-covariance matrix has eigenvalues equal to zero if and only if the variables are not linearly independent, it only remains to be shown that "if the matrix has eigenvalues equal to zero then the variables are not linearly independent".

If you have a zero eigenvalue for $C_{ij} = \text{Cov}(X_i,X_j)$ then there is some linear combination (defined by the eigenvector $v$)

$$Y = \sum_{i=1}^n v_i (X_i) $$

such that

$$\begin{array}{rcl} \text{Cov}(Y,Y) &=& \sum_{i=1}^n \sum_{j=1}^n v_i v_j \text{Cov}(X_i,X_j) \\ &=&\sum_{i=1}^n v_i\sum_{j=1}^n v_j C_{ij} \\ &= &\sum_{i=1}^n v_i \cdot 0 \\ &=& 0 \end{array}$$

which means that $Y$ needs to be a constant and thus the variables $X_i$ have to add up to a constant and are either constants themselves (the trivial case) or not linearly independent.

- the first line in the equation with $\text{Cov}(Y,Y)$ is due to the property of covariance $$\scriptsize\text{Cov}(aU+bV,cW+dX) = ac\,\text{Cov}(U,W) + bc\,\text{Cov}(V,W) +ad\, \text{Cov}(U,X) + bd \,\text{Cov}(V,X) $$

- the step from the second to the third line is due to the property of a zero eigenvalue $$\scriptsize \sum_{j=1}^nv_jC_{ij} = 0$$


Non-linear constraints

So, since linear constraints are a necessary condition (not just sufficient), non-linear constraints will only be relevant when they indirectly imply a (necessary) linear constraint.

In fact, there is a direct correspondence between the eigenvectors associated with the zero eigenvalue and the linear constraints.

$$C \cdot v = 0 \iff Y = \sum_{i=1}^n v_i X_i = \text{const}$$

Thus non-linear constraints leading to a zero eigenvalue must, together combined, generate some linear constraint.


How can non-linear constraints lead to linear constraints

Your example in the comments can show this intuitively how non-linear constraints can lead to linear constraints by reversing the derivation. The following non-linear constraints

$$\begin{array}{lcr} a^2+b^2&=&1\\ c^2+d^2&=&1\\ ac + bd &=& 0 \\ ad - bc &=& 1 \end{array}$$

can be reduced to

$$\begin{array}{lcr} a^2+b^2&=&1\\ c^2+d^2&=&1\\ a-d&=&0 \\ b+c &=& 0 \end{array}$$

You could inverse this. Say you have non-linear plus linear constraints, then it is not strange to imagine how we can replace one of the linear constraints with a non-linear constraint, by filling the linear constraints into the non-linear constraints. E.g when we substitute $a=d$ and $b=-c$ in the non-linear form $a^2+b^2=1$ then you can make another relationship $ad-bc=1$. And when you multiply $a=d$ and $c=-b$ then you get $ac=-bd$.

deleted 260 characters in body
Source Link
Sextus Empiricus
  • 86.5k
  • 6
  • 115
  • 302

###Linear independence is not just sufficient but also a neccesary condition

To show that the variance-covariance matrix has eigenvalues equal to zero if and only if the variables are not linearly independent, it only remains to be shown that "if the matrix has eigenvalues equal to zero then the variables are not linearly independent".

If you have a zero eigenvalue for $C_{ij} = \text{Cov}(X_i,X_j)$ then there is some linear combination (defined by the eigenvector $v$)

$$Y = \sum_{i=1}^n v_i (X_i) $$

such that

$$\begin{array}{rcl} \text{Cov}(Y,Y) &=& \sum_{i=1}^n \sum_{j=1}^n v_i v_j \text{Cov}(X_i,X_j) \\ &=&\sum_{i=1}^n v_i\sum_{j=1}^n v_j C_{ij} \\ &= &\sum_{i=1}^n v_i \cdot 0 \\ &=& 0 \end{array}$$

which means that $Y$ needs to be a constant and thus the variables $X_i$ have to add up to a constant and are either constants themselves (the trivial case) or not linearly independent.


###Notes about derivation

  • the first line in the equation with $\text{Cov}(Y,Y)$ is due to the property of covariance $$\scriptsize\text{Cov}(aU+bV,cW+dX) = ac\,\text{Cov}(U,W) + bc\,\text{Cov}(V,W) +ad\, \text{Cov}(U,X) + bd \,\text{Cov}(V,X) $$

  • the step from the second to the third line is due to the property of a zero eigenvalue $$\scriptsize \sum_{j=1}^nv_jC_{ij} = 0$$


Non-linear constraints

So, since linear constraints are a necessary condition (not just sufficient), non-linear constraints will only be relevant when they indirectly imply a (necessary) linear constraint.

In fact, there is a direct correspondence between the eigenvectors associated with the zero eigenvalue and the linear constraints.

$$C \cdot v = 0 \iff Y = \sum_{i=1}^n v_i X_i = \text{const}$$

Thus non-linear constraints leading to a zero eigenvalue must, together combined, generate some linear constraint.

Intuitively: The intersection of multiple non-linear constraints, can be seen as intersections of manifolds, which will only lead to a linear relationship (which represents a manifold with zero curvature) when the original manifolds have zero curvature (or when the intersections ends up as a set of points that can be placed on a hyper-plane).

Your example in the comments is an example where the intersections of non-linear constraints must lead to pointsbecome effectively a linear constraint.

With the matrix $$M=\begin{bmatrix} X_1 & X_2 \\ X_3 & X_4\end{bmatrix}$$ the constraint $M \cdot M^T = I$ can be translated to the variables parameterized as:

$$\begin{array}{rcl} X_1 &=& cos(\alpha) \\ X_2 &=& sin(\alpha) \\ X_3 &=& cos(\alpha +\frac{1+2k}{2} \pi) \\ X_4 &=& sin(\alpha +\frac{1+2k}{2} \pi) \\\end{array}$$$$\begin{array}{rcl} X_1 &=& cos(\alpha) \\ X_2 &=& sin(\alpha) \\ X_3 &=& sin(\alpha +(1+k) \pi) \\ X_4 &=& cos(\alpha +k \pi) \\\end{array}$$

a fourth constraint (although, like $\det M=1$ won't do it) may restrict, this surface/curve (which has non-zero curvature)reduces further to one or a few points.:

$$\begin{array}{rcrcr} X_1 &=& cos(\alpha) \\ X_2 &=& sin(\alpha) \\ X_3 &=& -sin(\alpha) &=& -X_2 \\ X_4 &=& cos(\alpha) &=& X_1 \\\end{array}$$

###Linear independence is not just sufficient but also a neccesary condition

To show that the variance-covariance matrix has eigenvalues equal to zero if and only if the variables are not linearly independent, it only remains to be shown that "if the matrix has eigenvalues equal to zero then the variables are not linearly independent".

If you have a zero eigenvalue for $C_{ij} = \text{Cov}(X_i,X_j)$ then there is some linear combination (defined by the eigenvector $v$)

$$Y = \sum_{i=1}^n v_i (X_i) $$

such that

$$\begin{array}{rcl} \text{Cov}(Y,Y) &=& \sum_{i=1}^n \sum_{j=1}^n v_i v_j \text{Cov}(X_i,X_j) \\ &=&\sum_{i=1}^n v_i\sum_{j=1}^n v_j C_{ij} \\ &= &\sum_{i=1}^n v_i \cdot 0 \\ &=& 0 \end{array}$$

which means that $Y$ needs to be a constant and thus the variables $X_i$ have to add up to a constant and are either constants themselves (the trivial case) or not linearly independent.


###Notes about derivation

  • the first line in the equation with $\text{Cov}(Y,Y)$ is due to the property of covariance $$\scriptsize\text{Cov}(aU+bV,cW+dX) = ac\,\text{Cov}(U,W) + bc\,\text{Cov}(V,W) +ad\, \text{Cov}(U,X) + bd \,\text{Cov}(V,X) $$

  • the step from the second to the third line is due to the property of a zero eigenvalue $$\scriptsize \sum_{j=1}^nv_jC_{ij} = 0$$


Non-linear constraints

So, since linear constraints are a necessary condition (not just sufficient), non-linear constraints will only be relevant when they indirectly imply a (necessary) linear constraint.

In fact, there is a direct correspondence between the eigenvectors associated with the zero eigenvalue and the linear constraints.

$$C \cdot v = 0 \iff Y = \sum_{i=1}^n v_i X_i = \text{const}$$

Thus non-linear constraints leading to a zero eigenvalue must, together combined, generate some linear constraint.

Intuitively: The intersection of multiple non-linear constraints, can be seen as intersections of manifolds, which will only lead to a linear relationship (which represents a manifold with zero curvature) when the original manifolds have zero curvature (or when the intersections ends up as a set of points that can be placed on a hyper-plane).

Your example in the comments is an example where the non-linear constraints must lead to points.

With the matrix $$M=\begin{bmatrix} X_1 & X_2 \\ X_3 & X_4\end{bmatrix}$$ the constraint $M \cdot M^T = I$ can be translated to the variables parameterized as:

$$\begin{array}{rcl} X_1 &=& cos(\alpha) \\ X_2 &=& sin(\alpha) \\ X_3 &=& cos(\alpha +\frac{1+2k}{2} \pi) \\ X_4 &=& sin(\alpha +\frac{1+2k}{2} \pi) \\\end{array}$$

a fourth constraint (although $\det M=1$ won't do it) may restrict this surface/curve (which has non-zero curvature) to one or a few points.

###Linear independence is not just sufficient but also a neccesary condition

To show that the variance-covariance matrix has eigenvalues equal to zero if and only if the variables are not linearly independent, it only remains to be shown that "if the matrix has eigenvalues equal to zero then the variables are not linearly independent".

If you have a zero eigenvalue for $C_{ij} = \text{Cov}(X_i,X_j)$ then there is some linear combination (defined by the eigenvector $v$)

$$Y = \sum_{i=1}^n v_i (X_i) $$

such that

$$\begin{array}{rcl} \text{Cov}(Y,Y) &=& \sum_{i=1}^n \sum_{j=1}^n v_i v_j \text{Cov}(X_i,X_j) \\ &=&\sum_{i=1}^n v_i\sum_{j=1}^n v_j C_{ij} \\ &= &\sum_{i=1}^n v_i \cdot 0 \\ &=& 0 \end{array}$$

which means that $Y$ needs to be a constant and thus the variables $X_i$ have to add up to a constant and are either constants themselves (the trivial case) or not linearly independent.


###Notes about derivation

  • the first line in the equation with $\text{Cov}(Y,Y)$ is due to the property of covariance $$\scriptsize\text{Cov}(aU+bV,cW+dX) = ac\,\text{Cov}(U,W) + bc\,\text{Cov}(V,W) +ad\, \text{Cov}(U,X) + bd \,\text{Cov}(V,X) $$

  • the step from the second to the third line is due to the property of a zero eigenvalue $$\scriptsize \sum_{j=1}^nv_jC_{ij} = 0$$


Non-linear constraints

So, since linear constraints are a necessary condition (not just sufficient), non-linear constraints will only be relevant when they indirectly imply a (necessary) linear constraint.

In fact, there is a direct correspondence between the eigenvectors associated with the zero eigenvalue and the linear constraints.

$$C \cdot v = 0 \iff Y = \sum_{i=1}^n v_i X_i = \text{const}$$

Thus non-linear constraints leading to a zero eigenvalue must, together combined, generate some linear constraint.

Your example in the comments is an example where the intersections of non-linear constraints become effectively a linear constraint.

With the matrix $$M=\begin{bmatrix} X_1 & X_2 \\ X_3 & X_4\end{bmatrix}$$ the constraint $M \cdot M^T = I$ can be translated to the variables parameterized as:

$$\begin{array}{rcl} X_1 &=& cos(\alpha) \\ X_2 &=& sin(\alpha) \\ X_3 &=& sin(\alpha +(1+k) \pi) \\ X_4 &=& cos(\alpha +k \pi) \\\end{array}$$

a fourth constraint, like $\det M=1$, this reduces further to:

$$\begin{array}{rcrcr} X_1 &=& cos(\alpha) \\ X_2 &=& sin(\alpha) \\ X_3 &=& -sin(\alpha) &=& -X_2 \\ X_4 &=& cos(\alpha) &=& X_1 \\\end{array}$$

added 4 characters in body
Source Link
Sextus Empiricus
  • 86.5k
  • 6
  • 115
  • 302

###Linear independence is not just sufficient but also a neccesary condition

To show that the variance-covariance matrix has eigenvalues equal to zero if and only if the variables are not linearly independent, it only remains to be shown that "if the matrix has eigenvalues equal to zero then the variables are not linearly independent".

If you have a zero eigenvalue for $C_{ij} = \text{Cov}(X_i,X_j)$ then there is some linear combination (defined by the eigenvector $v$)

$$Y = \sum_{i=1}^n v_i (X_i) $$

such that

$$\begin{array}{rcl} \text{Cov}(Y,Y) &=& \sum_{i=1}^n \sum_{j=1}^n v_i v_j \text{Cov}(X_i,X_j) \\ &=&\sum_{i=1}^n v_i\sum_{j=1}^n v_j C_{ij} \\ &= &\sum_{i=1}^n v_i \cdot 0 \\ &=& 0 \end{array}$$

which means that $Y$ needs to be a constant and thus the variables $X_i$ have to add up to a constant and are either constants themselves (the trivial case) or not linearly independent.


###Notes about derivation

  • the first line in the equation with $\text{Cov}(Y,Y)$ is due to the property of covariance $$\scriptsize\text{Cov}(aU+bV,cW+dX) = ac\,\text{Cov}(U,W) + bc\,\text{Cov}(V,W) +ad\, \text{Cov}(U,X) + bd \,\text{Cov}(V,X) $$

  • the step from the second to the third line is due to the property of a zero eigenvalue $$\scriptsize \sum_{j=1}^nv_jC_{ij} = 0$$


Non-linear constraints

So, since linear constraints are a necessary condition (not just sufficient), non-linear constraints will only be relevant when they indirectly imply a (necessary) linear constraint.

In fact, there is a direct correspondence between the eigenvectors associated with the zero eigenvalue and the linear constraints.

$$C \cdot v = 0 \iff Y = \sum_{i=1}^n v_i X_i = \text{const}$$

Thus non-linear constraints leading to a zero eigenvalue must, together combined, generate some linear constraint.

Intuitively: The intersection of multiple non-linear constraints, can be seen as intersections of manifolds, which will only lead to a linear relationship (which represents a manifold with zero curvature) when the original manifolds have zero curvature (or when the intersections ends up as a set of points that can be placed on a hyper-plane).

Your example in the comments is an example where the non-linear constraints must lead to points.

With the matrix $$M=\begin{bmatrix} X_1 & X_2 \\ X_3 & X_4\end{bmatrix}$$ the constraint $M \cdot M^T$$M \cdot M^T = I$ can be translated to the variables parameterized as:

$$\begin{array}{rcl} X_1 &=& cos(\alpha) \\ X_2 &=& sin(\alpha) \\ X_3 &=& cos(\alpha +\frac{1+2k}{2} \pi) \\ X_4 &=& sin(\alpha +\frac{1+2k}{2} \pi) \\\end{array}$$

a fourth constraint (although $\det M=1$ won't do it) may restrict this surface/curve (which has non-zero curvature) to one or a few points.

###Linear independence is not just sufficient but also a neccesary condition

To show that the variance-covariance matrix has eigenvalues equal to zero if and only if the variables are not linearly independent, it only remains to be shown that "if the matrix has eigenvalues equal to zero then the variables are not linearly independent".

If you have a zero eigenvalue for $C_{ij} = \text{Cov}(X_i,X_j)$ then there is some linear combination (defined by the eigenvector $v$)

$$Y = \sum_{i=1}^n v_i (X_i) $$

such that

$$\begin{array}{rcl} \text{Cov}(Y,Y) &=& \sum_{i=1}^n \sum_{j=1}^n v_i v_j \text{Cov}(X_i,X_j) \\ &=&\sum_{i=1}^n v_i\sum_{j=1}^n v_j C_{ij} \\ &= &\sum_{i=1}^n v_i \cdot 0 \\ &=& 0 \end{array}$$

which means that $Y$ needs to be a constant and thus the variables $X_i$ have to add up to a constant and are either constants themselves (the trivial case) or not linearly independent.


###Notes about derivation

  • the first line in the equation with $\text{Cov}(Y,Y)$ is due to the property of covariance $$\scriptsize\text{Cov}(aU+bV,cW+dX) = ac\,\text{Cov}(U,W) + bc\,\text{Cov}(V,W) +ad\, \text{Cov}(U,X) + bd \,\text{Cov}(V,X) $$

  • the step from the second to the third line is due to the property of a zero eigenvalue $$\scriptsize \sum_{j=1}^nv_jC_{ij} = 0$$


Non-linear constraints

So, since linear constraints are a necessary condition (not just sufficient), non-linear constraints will only be relevant when they indirectly imply a (necessary) linear constraint.

In fact, there is a direct correspondence between the eigenvectors associated with the zero eigenvalue and the linear constraints.

$$C \cdot v = 0 \iff Y = \sum_{i=1}^n v_i X_i = \text{const}$$

Thus non-linear constraints leading to a zero eigenvalue must, together combined, generate some linear constraint.

Intuitively: The intersection of multiple non-linear constraints, can be seen as intersections of manifolds, which will only lead to a linear relationship (which represents a manifold with zero curvature) when the original manifolds have zero curvature (or when the intersections ends up as a set of points that can be placed on a hyper-plane).

Your example in the comments is an example where the non-linear constraints must lead to points.

With the matrix $$M=\begin{bmatrix} X_1 & X_2 \\ X_3 & X_4\end{bmatrix}$$ the constraint $M \cdot M^T$ can be translated to the variables parameterized as:

$$\begin{array}{rcl} X_1 &=& cos(\alpha) \\ X_2 &=& sin(\alpha) \\ X_3 &=& cos(\alpha +\frac{1+2k}{2} \pi) \\ X_4 &=& sin(\alpha +\frac{1+2k}{2} \pi) \\\end{array}$$

a fourth constraint (although $\det M=1$ won't do it) may restrict this surface/curve (which has non-zero curvature) to one or a few points.

###Linear independence is not just sufficient but also a neccesary condition

To show that the variance-covariance matrix has eigenvalues equal to zero if and only if the variables are not linearly independent, it only remains to be shown that "if the matrix has eigenvalues equal to zero then the variables are not linearly independent".

If you have a zero eigenvalue for $C_{ij} = \text{Cov}(X_i,X_j)$ then there is some linear combination (defined by the eigenvector $v$)

$$Y = \sum_{i=1}^n v_i (X_i) $$

such that

$$\begin{array}{rcl} \text{Cov}(Y,Y) &=& \sum_{i=1}^n \sum_{j=1}^n v_i v_j \text{Cov}(X_i,X_j) \\ &=&\sum_{i=1}^n v_i\sum_{j=1}^n v_j C_{ij} \\ &= &\sum_{i=1}^n v_i \cdot 0 \\ &=& 0 \end{array}$$

which means that $Y$ needs to be a constant and thus the variables $X_i$ have to add up to a constant and are either constants themselves (the trivial case) or not linearly independent.


###Notes about derivation

  • the first line in the equation with $\text{Cov}(Y,Y)$ is due to the property of covariance $$\scriptsize\text{Cov}(aU+bV,cW+dX) = ac\,\text{Cov}(U,W) + bc\,\text{Cov}(V,W) +ad\, \text{Cov}(U,X) + bd \,\text{Cov}(V,X) $$

  • the step from the second to the third line is due to the property of a zero eigenvalue $$\scriptsize \sum_{j=1}^nv_jC_{ij} = 0$$


Non-linear constraints

So, since linear constraints are a necessary condition (not just sufficient), non-linear constraints will only be relevant when they indirectly imply a (necessary) linear constraint.

In fact, there is a direct correspondence between the eigenvectors associated with the zero eigenvalue and the linear constraints.

$$C \cdot v = 0 \iff Y = \sum_{i=1}^n v_i X_i = \text{const}$$

Thus non-linear constraints leading to a zero eigenvalue must, together combined, generate some linear constraint.

Intuitively: The intersection of multiple non-linear constraints, can be seen as intersections of manifolds, which will only lead to a linear relationship (which represents a manifold with zero curvature) when the original manifolds have zero curvature (or when the intersections ends up as a set of points that can be placed on a hyper-plane).

Your example in the comments is an example where the non-linear constraints must lead to points.

With the matrix $$M=\begin{bmatrix} X_1 & X_2 \\ X_3 & X_4\end{bmatrix}$$ the constraint $M \cdot M^T = I$ can be translated to the variables parameterized as:

$$\begin{array}{rcl} X_1 &=& cos(\alpha) \\ X_2 &=& sin(\alpha) \\ X_3 &=& cos(\alpha +\frac{1+2k}{2} \pi) \\ X_4 &=& sin(\alpha +\frac{1+2k}{2} \pi) \\\end{array}$$

a fourth constraint (although $\det M=1$ won't do it) may restrict this surface/curve (which has non-zero curvature) to one or a few points.

added 311 characters in body
Source Link
Sextus Empiricus
  • 86.5k
  • 6
  • 115
  • 302
Loading
added 311 characters in body
Source Link
Sextus Empiricus
  • 86.5k
  • 6
  • 115
  • 302
Loading
added 39 characters in body
Source Link
Sextus Empiricus
  • 86.5k
  • 6
  • 115
  • 302
Loading
added 8 characters in body
Source Link
Sextus Empiricus
  • 86.5k
  • 6
  • 115
  • 302
Loading
added 8 characters in body
Source Link
Sextus Empiricus
  • 86.5k
  • 6
  • 115
  • 302
Loading
added 35 characters in body
Source Link
Sextus Empiricus
  • 86.5k
  • 6
  • 115
  • 302
Loading
added 1156 characters in body
Source Link
Sextus Empiricus
  • 86.5k
  • 6
  • 115
  • 302
Loading
added 18 characters in body
Source Link
Sextus Empiricus
  • 86.5k
  • 6
  • 115
  • 302
Loading
added 63 characters in body
Source Link
Sextus Empiricus
  • 86.5k
  • 6
  • 115
  • 302
Loading
added 63 characters in body
Source Link
Sextus Empiricus
  • 86.5k
  • 6
  • 115
  • 302
Loading
added 25 characters in body
Source Link
Sextus Empiricus
  • 86.5k
  • 6
  • 115
  • 302
Loading
added 25 characters in body
Source Link
Sextus Empiricus
  • 86.5k
  • 6
  • 115
  • 302
Loading
Source Link
Sextus Empiricus
  • 86.5k
  • 6
  • 115
  • 302
Loading