PCA vs linear Autoencoder: features independence

Question

Principal component analysis is a technique that extract the best orthogonal subspace in which we can project our points with less information loss, maximizing the variance.

A linear auto encoder is a neural network composed by an encoder (single layer) that compresses our space in a new subspace, which is not necessarily orthogonal, and of a decoder that reconstruct our data with less information loss possible.

In substance, both the models are capable of features reduction, by projecting the original space in a new optimal subspace with and without a constraint of orthogonality.

In this publication in which is explained how Variational Autoencoders works, when PCA and linear autoencoder are compared, is stated that:

[...] Indeed, several basis can be chosen to describe the same optimal subspace and, so, several encoder/decoder pairs can give the optimal reconstruction error. Moreover, for linear autoencoders and contrarily to PCA, the new features we end up do not have to be independent (no orthogonality constraints in the neural networks). [...]

Why if I project my points in a subspace that has no orthogonality constraints, my features end up to be not necessarily independent? And why in the orthogonal space the new features, linear composition of the previous one, are assumed to be independent?

Answer 1

The answer turns on the definitions of orthogonal and linearly independent vectors. They're distinct concepts.

The reasoning of the author is if a set of vectors is an orthogonal set, then it also linearly independent. Here's a simple proof from https://sites.math.rutgers.edu/~cherlin/Courses/250/Lectures/250L23.html

Theorem Any orthogonal set of vectors is linearly independent.

To see this result, suppose that $v_1, . . ., v_k$ are in this orthogonal set, and there are constants $c_1, . . ., c_k$ such that $c_1 v_1 + · · · + c_k v_k = 0$. For any $j$ between $1$ and $k$, take the dot product of $v_j$ with both sides of this equation. We obtain $c_j \|v_j \|^2 = 0$, and since $v_j$ is not 0 (otherwise the set could not be orthogonal), this forces $c_j = 0$. Thus the only linear combinations of vectors in the set which equal the 0 vector are those in which all of the coefficients are zero, which means that the set is linearly independent.

The linear autoencoders in your question are not constrained to have an orthogonal basis, so we can't rely on this theorem when reasoning about the linear independence of the autoencoder's output. Without guaranteed orthogonality, the autoencoder might or might not yield a set of linearly independent vectors.

Importantly, a set of vectors may be non-orthogonal yet still be linearly independent. Here's an example. The set of vectors $$ v_1 =\begin{bmatrix}{1 \\ 1}\end{bmatrix}, v_2 =\begin{bmatrix}{-3 \\ 2}\end{bmatrix} $$ is linearly independent. However, they are not orthogonal because the dot product is nonzero.

Answer 2

On the question: "Why if I project my points in a subspace that has no orthogonality constraints, my features end up to be not necessarily independent?, per the theorem (already cited): "Any orthogonal set of vectors is linearly independent", it follows orthogonality also implies linearly independent.

However, a source cited above notes that "the new features we end up do not have to be independent (no orthogonality constraints in the neural networks)", so the new features data set is neither orthogonal or even [EDIT] necessarily [END EDIT] linearly independent. So, this topic's very title question: "PCA versus Linear Autoencoder: features independence", appears to be [EDIT] possibly [END EDIT] problematic as there is [EDIT] necessarily [END EDIT] no 'independence', at least, in a Linear Algebra sense.

On Principal Component Analysis (PCA), per a source, to quote:

Given a collection of points in two, three, or higher dimensional space, a "best fitting" line can be defined as one that minimizes the average squared distance from a point to the line. The next best-fitting line can be similarly chosen from directions perpendicular to the first. Repeating this process yields an orthogonal basis in which different individual dimensions of the data are uncorrelated. These basis vectors are called principal components, and several related procedures principal component analysis (PCA).

And, importantly related to applications:

PCA is mostly used as a tool in exploratory data analysis and for making predictive models. It is often used to visualize genetic distance and relatedness between populations.

So the referred to data reduction construct referred to as 'features independence' may result in data consolidation, but it, I would argue, in a comparative sense to PCA, does not readily foster facile paths to either exploratory data analysis or statistical-based predictions.

[EDIT] Further, with respect autoencoder, some background material provided by Wikipedia displaying its usefulness in various areas, clearly diverging from PCA, to quote:

An autoencoder is a type of artificial neural network used to learn efficient data codings in an unsupervised manner.[1] The aim of an autoencoder is to learn a representation (encoding) for a set of data, typically for dimensionality reduction, by training the network to ignore signal “noise”. Along with the reduction side, a reconstructing side is learnt, where the autoencoder tries to generate from the reduced encoding a representation as close as possible to its original input, hence its name. Several variants exist to the basic model, with the aim of forcing the learned representations of the input to assume useful properties.[2]...Autoencoders are effectively used for solving many applied problems, from face recognition[5] to acquiring the semantic meaning of words.[6][7]