Principal component analysis is a technique that extract the best orthogonal subspace in which we can project our points with less information loss, maximizing the variance.

A linear auto encoder is a neural network composed by an encoder (single layer) that compresses our space in a new subspace, which is not necessarily orthogonal, and of a decoder that reconstruct our data with less information loss possible.

In substance, both the models are capable of features reduction, by projecting the original space in a new optimal subspace with and without a constraint of orthogonality.

In this publication in which is explained how Variational Autoencoders works, when PCA and linear autoencoder are compared, is stated that:

[...] Indeed, several basis can be chosen to describe the same optimal subspace and, so, several encoder/decoder pairs can give the optimal reconstruction error. Moreover, for linear autoencoders and contrarily to PCA, the new features we end up do not have to be independent (no orthogonality constraints in the neural networks). [...]

Why if I project my points in a subspace that has no orthogonality constraints, my features end up to be not necessarily independent? And why in the orthogonal space the new features, linear composition of the previous one, are assumed to be independent?

PCA vs linear Autoencoder


The answer turns on the definitions of orthogonal and linearly independent vectors. They're distinct concepts.

The reasoning of the author is if a set of vectors is an orthogonal set, then it also linearly independent. Here's a simple proof from https://sites.math.rutgers.edu/~cherlin/Courses/250/Lectures/250L23.html

Theorem Any orthogonal set of vectors is linearly independent.

To see this result, suppose that $v_1, . . ., v_k$ are in this orthogonal set, and there are constants $c_1, . . ., c_k$ such that $c_1 v_1 + · · · + c_k v_k = 0$. For any $j$ between $1$ and $k$, take the dot product of $v_j$ with both sides of this equation. We obtain $c_j \|v_j \|^2 = 0$, and since $v_j$ is not 0 (otherwise the set could not be orthogonal), this forces $c_j = 0$. Thus the only linear combinations of vectors in the set which equal the 0 vector are those in which all of the coefficients are zero, which means that the set is linearly independent.

The linear autoencoders in your question are not constrained to have an orthogonal basis, so we can't rely on this theorem when reasoning about the linear independence of the autoencoder's output. Without guaranteed orthogonality, the autoencoder might or might not yield a set of linearly independent vectors.

Importantly, a set of vectors may be non-orthogonal yet still be linearly independent. Here's an example. The set of vectors $$ v_1 =\begin{bmatrix}{1 \\ 1}\end{bmatrix}, v_2 =\begin{bmatrix}{-3 \\ 2}\end{bmatrix} $$ is linearly independent. However, they are not orthogonal because the dot product is nonzero.

  • $\begingroup$ My problem is that I confused the mathematical linear independence with the probabilistic independence $\endgroup$ – Nikaido May 27 '20 at 18:18
  • 1
    $\begingroup$ @Nikaido, Oh, yes, that is a confusing overlap of terminology! $\endgroup$ – Sycorax May 27 '20 at 18:20

On the question: "Why if I project my points in a subspace that has no orthogonality constraints, my features end up to be not necessarily independent?, per the theorem (already cited): "Any orthogonal set of vectors is linearly independent", it follows orthogonality also implies linearly independent.

However, a source cited above notes that "the new features we end up do not have to be independent (no orthogonality constraints in the neural networks)", so the new features data set is neither orthogonal or even [EDIT] necessarily [END EDIT] linearly independent. So, this topic's very title question: "PCA versus Linear Autoencoder: features independence", appears to be [EDIT] possibly [END EDIT] problematic as there is [EDIT] necessarily [END EDIT] no 'independence', at least, in a Linear Algebra sense.

On Principal Component Analysis (PCA), per a source, to quote:

Given a collection of points in two, three, or higher dimensional space, a "best fitting" line can be defined as one that minimizes the average squared distance from a point to the line. The next best-fitting line can be similarly chosen from directions perpendicular to the first. Repeating this process yields an orthogonal basis in which different individual dimensions of the data are uncorrelated. These basis vectors are called principal components, and several related procedures principal component analysis (PCA).

And, importantly related to applications:

PCA is mostly used as a tool in exploratory data analysis and for making predictive models. It is often used to visualize genetic distance and relatedness between populations.

So the referred to data reduction construct referred to as 'features independence' may result in data consolidation, but it, I would argue, in a comparative sense to PCA, does not readily foster facile paths to either exploratory data analysis or statistical-based predictions.

[EDIT] Further, with respect autoencoder, some background material provided by Wikipedia displaying its usefulness in various areas, clearly diverging from PCA, to quote:

An autoencoder is a type of artificial neural network used to learn efficient data codings in an unsupervised manner.[1] The aim of an autoencoder is to learn a representation (encoding) for a set of data, typically for dimensionality reduction, by training the network to ignore signal “noise”. Along with the reduction side, a reconstructing side is learnt, where the autoencoder tries to generate from the reduced encoding a representation as close as possible to its original input, hence its name. Several variants exist to the basic model, with the aim of forcing the learned representations of the input to assume useful properties.[2]...Autoencoders are effectively used for solving many applied problems, from face recognition[5] to acquiring the semantic meaning of words.[6][7]

  • 1
    $\begingroup$ I think you're misreading the quotation. "[T]he new features we end up do not have to be independent" means that the new features might be independent or might not be independent, but instead you write "the new features data set is neither orthogonal or even linearly independent." It's true that they're not orthogonal, but vectors can be linearly independent and also be non-orthogonal. An example is shown in my answer. $\endgroup$ – Sycorax May 28 '20 at 15:18
  • $\begingroup$ Sycorax: I will accept your point, and have clearly placed Edit and [END EDIT] in my answer. I doubt, however, in my opinion, as to whether it significantly distracts from the key point of my analysis. Supporting my opinion, I have further added an extract from Wikipedia as provided background . $\endgroup$ – AJKOER May 28 '20 at 18:41
  • $\begingroup$ Your third edit states "...there is necessarily no 'independence', at least, in a Linear Algebra sense," which appears to suggest that the resulting feature vectors are not linearly independent as a matter of necessity. I think you meant to write "there is not necessarily independence in a linear algebra sense," which means that the result may or may not be linear independent vectors. $\endgroup$ – Sycorax May 28 '20 at 18:45
  • $\begingroup$ Sycorax: Agree, clumsy, but trying to edit for correctness and maintain original words. I also added some of the applications' success stories as background differentiation from PCA. $\endgroup$ – AJKOER May 28 '20 at 18:57
  • $\begingroup$ Keep in mind that the edit history of the post will always be available, so you don't have to worry about discarding what you originally wrote. If someone wants to see earlier versions, they can find it easily using the edit history. Just a tip. $\endgroup$ – Sycorax May 28 '20 at 18:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.