Return to Answer

The answer turns on the definitions of orthogonal and linearly independent vectors. They're distinct concepts.

The reasoning of the author is if a set of vectors is an orthogonal set, then it also linearly independent. Here's a simple proof from https://sites.math.rutgers.edu/~cherlin/Courses/250/Lectures/250L23.html

Theorem Any orthogonal set of vectors is linearly independent.

 

To see this result, suppose that $v_1, . . ., v_k$ are in this orthogonal set, and there are constants $c_1, . . ., c_k$ such that $c_1 v_1 + · · · + c_k v_k = 0$. For any $j$ between $1$ and $k$, take the dot product of $v_j$ with both sides of this equation. We obtain $c_j \|v_j \|^2 = 0$, and since $v_j$ is not 0 (otherwise the set could not be orthogonal), this forces $c_j = 0$. Thus the only linear combinations of vectors in the set which equal the 0 vector are those in which all of the coefficients are zero, which means that the set is linearly independent.

The linear autoencoders in your question are not constrained to have an orthogonal basis, so we can't rely on this theorem when reasoning about the linear independence of the autoencoder's output. Without guaranteed orthogonality, the autoencoder might or might not yield a set of linearly independent vectors.

Importantly, a set of vectors may be non-orthogonal yet still be linearly independent. Here's an example. The set of vectors $$ v_1 =\begin{bmatrix}{1 \\ 1}\end{bmatrix}, v_2 =\begin{bmatrix}{-3 \\ 2}\end{bmatrix} $$ is linearly independent. However, they are not orthogonal because the dot product is nonzero.

The answer turns on the definitions of orthogonal and linearly independent vectors. They're distinct concepts.

The reasoning of the author is if a set of vectors is an orthogonal set, then it also linearly independent. Here's a simple proof from https://sites.math.rutgers.edu/~cherlin/Courses/250/Lectures/250L23.html

Theorem Any orthogonal set of vectors is linearly independent.

To see this result, suppose that $v_1, . . ., v_k$ are in this orthogonal set, and there are constants $c_1, . . ., c_k$ such that $c_1 v_1 + · · · + c_k v_k = 0$. For any $j$ between $1$ and $k$, take the dot product of $v_j$ with both sides of this equation. We obtain $c_j \|v_j \|^2 = 0$, and since $v_j$ is not 0 (otherwise the set could not be orthogonal), this forces $c_j = 0$. Thus the only linear combinations of vectors in the set which equal the 0 vector are those in which all of the coefficients are zero, which means that the set is linearly independent.

The linear autoencoders in your question are not constrained to have an orthogonal basis, so we can't rely on this theorem when reasoning about the linear independence of the autoencoder's output. Without guaranteed orthogonality, the autoencoder might or might not yield a set of linearly independent vectors.

Importantly, a set of vectors may be non-orthogonal yet still be linearly independent. Here's an example. The set of vectors $$ v_1 =\begin{bmatrix}{1 \\ 1}\end{bmatrix}, v_2 =\begin{bmatrix}{-3 \\ 2}\end{bmatrix} $$ is linearly independent. However, they are not orthogonal because the dot product is nonzero.

The answer turns on the definitions of orthogonal and linearly independent vectors. They're distinct concepts.

The reasoning of the author is if a set of vectors is an orthogonal set, then it also linearly independent. Here's a simple proof from https://sites.math.rutgers.edu/~cherlin/Courses/250/Lectures/250L23.html

Theorem Any orthogonal set of vectors is linearly independent.

To see this result, suppose that $v_1, . . ., v_k$ are in this orthogonal set, and there are constants $c_1, . . ., c_k$ such that $c_1 v_1 + · · · + c_k v_k = 0$. For any $j$ between $1$ and $k$, take the dot product of $v_j$ with both sides of this equation. We obtain $c_j \|v_j \|^2 = 0$, and since $v_j$ is not 0 (otherwise the set could not be orthogonal), this forces $c_j = 0$. Thus the only linear combinations of vectors in the set which equal the 0 vector are those in which all of the coefficients are zero, which means that the set is linearly independent.

Giving up the orthogonality constraint implies that we can't rely on this proof when reasoning about the linear independence of the autoencoder's output. Without guaranteed orthogonality, the autoencoder might or might not yield a set of linearly independent vectors.

Importantly, a set of vectors may be non-orthogonal yet still be linearly independent. Here's an example. The set of vectors $$ v_1 =\begin{bmatrix}{1 \\ 1}\end{bmatrix}, v_2 =\begin{bmatrix}{-3 \\ 2}\end{bmatrix} $$ is linearly independent. However, they are not orthogonal because the dot product is nonzero.

The answer turns on the definitions of orthogonal and linearly independent vectors. They're distinct concepts.

The reasoning of the author is if a set of vectors is an orthogonal set, then it also linearly independent. Here's a simple proof from https://sites.math.rutgers.edu/~cherlin/Courses/250/Lectures/250L23.html

Theorem Any orthogonal set of vectors is linearly independent.

To see this result, suppose that $v_1, . . ., v_k$ are in this orthogonal set, and there are constants $c_1, . . ., c_k$ such that $c_1 v_1 + · · · + c_k v_k = 0$. For any $j$ between $1$ and $k$, take the dot product of $v_j$ with both sides of this equation. We obtain $c_j \|v_j \|^2 = 0$, and since $v_j$ is not 0 (otherwise the set could not be orthogonal), this forces $c_j = 0$. Thus the only linear combinations of vectors in the set which equal the 0 vector are those in which all of the coefficients are zero, which means that the set is linearly independent.

The linear autoencoders in your question are not constrained to have an orthogonal basis, so we can't rely on this theorem when reasoning about the linear independence of the autoencoder's output. Without guaranteed orthogonality, the autoencoder might or might not yield a set of linearly independent vectors.

Importantly, a set of vectors may be non-orthogonal yet still be linearly independent. Here's an example. The set of vectors $$ v_1 =\begin{bmatrix}{1 \\ 1}\end{bmatrix}, v_2 =\begin{bmatrix}{-3 \\ 2}\end{bmatrix} $$ is linearly independent. However, they are not orthogonal because the dot product is nonzero.

The answer turns on the definitions of orthogonal and linearly independent vectors. They're distinct concepts.

The reasoning of the author is if a set of vectors is an orthogonal set, then it also linearly independent. Here's a simple proof from https://sites.math.rutgers.edu/~cherlin/Courses/250/Lectures/250L23.html

Theorem Any orthogonal set of vectors is linearly independent.

To see this result, suppose that $v_1, . . ., v_k$ are in this orthogonal set, and there are constants $c_1, . . ., c_k$ such that $c_1 v_1 + · · · + c_k v_k = 0$. For any $j$ between $1$ and $k$, take the dot product of $v_j$ with both sides of this equation. We obtain $c_j \|v_j \|^2 = 0$, and since $v_j$ is not 0 (otherwise the set could not be orthogonal), this forces $c_j = 0$. Thus the only linear combinations of vectors in the set which equal the 0 vector are those in which all of the coefficients are zero, which means that the set is linearly independent.

Giving up the orthogonality constraint implies that we can't rely on this proof when reasoning about the linear independence of the autoencoder's output. Without guaranteed orthogonality, the autoencoder might or might not yield a set of linearly independent vectors.

Importantly, a set of vectors may be non-orthogonal yet still be linearly independent. Here's an example. The set of vectors $$ v_1 =\begin{bmatrix}{1 \\ 1}\end{bmatrix}, v_2 =\begin{bmatrix}{-3 \\ 2}\end{bmatrix}, v_3 =\begin{bmatrix}{2 \\ 4}\end{bmatrix} $$ is linearly dependent. However, they are not orthogonal because the dot products of each pair are nonzero.

The answer turns on the definitions of orthogonal and linearly independent vectors. They're distinct concepts.

The reasoning of the author is if a set of vectors is an orthogonal set, then it also linearly independent. Here's a simple proof from https://sites.math.rutgers.edu/~cherlin/Courses/250/Lectures/250L23.html

Theorem Any orthogonal set of vectors is linearly independent.

To see this result, suppose that $v_1, . . ., v_k$ are in this orthogonal set, and there are constants $c_1, . . ., c_k$ such that $c_1 v_1 + · · · + c_k v_k = 0$. For any $j$ between $1$ and $k$, take the dot product of $v_j$ with both sides of this equation. We obtain $c_j \|v_j \|^2 = 0$, and since $v_j$ is not 0 (otherwise the set could not be orthogonal), this forces $c_j = 0$. Thus the only linear combinations of vectors in the set which equal the 0 vector are those in which all of the coefficients are zero, which means that the set is linearly independent.

Giving up the orthogonality constraint implies that we can't rely on this proof when reasoning about the linear independence of the autoencoder's output. Without guaranteed orthogonality, the autoencoder might or might not yield a set of linearly independent vectors.

Importantly, a set of vectors may be non-orthogonal yet still be linearly independent. Here's an example. The set of vectors $$ v_1 =\begin{bmatrix}{1 \\ 1}\end{bmatrix}, v_2 =\begin{bmatrix}{-3 \\ 2}\end{bmatrix} $$ is linearly independent. However, they are not orthogonal because the dot product is nonzero.