Writing PCA as a special kind of auto-encoder

Question

I would like to know if it's possible to view PCA as a particular type of neural network, however, there's one major stumbling block that I haven't yet been able to get past.

Define the following "single-layer neural network with linear transfer/activation function": $$\mathrm{nn}(x) = Wx + b.$$ What loss function or training scheme would you use in order to get $\mathrm{nn}$ to behave as PCA? Specifically, is there a gradient descent scheme that seems to guarantee, at least in practice, the orthonormality of the rows of $W$? PCA gives the transformation $$ X \mapsto U\Sigma $$ defined by $$ f(\tilde X) = \tilde X V $$ where $X = U\Sigma V^*$ is the singular value decomposition of the data matrix $X$.

Note: The way I've written it, $x$ is a column vector, and so $x^T$ is the $j$th row ("observation") of the matrix $X$. So, it must be that $W = V^T$ and either $b \equiv 0$ [if there is no whitening], or $b = V^T \mu$ where $\mu$ is the vector of column means of $X$ [if there is whitening].

Answer 1

Orthogonality of the components is not something that you can back-propagate. You would have to change the optimization to include a step that also enforces orthogonality after each update. This is expensive, and the result may not be everywhere differentiable. (This is discussed in passing in "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift", Sergey Ioffe and Christian Szegedy)

However, you can use a linear auto-encoder to discover a basis that spans the principle components, or a non-linear auto-encoder to have a similar analogue to nonlinear PCA.

More information can be found in "From Principal Subspaces to Principal Components with Linear Autoencoders" by Elad Plaut.

The auto-encoder is an effective unsupervised learning model which is widely used in deep learning. It is well known that an auto-encoder with a single fully-connected hidden layer, a linear activation function and a squared error cost function trains weights that span the same subspace as the one spanned by the principal component loading vectors, but that they are not identical to the loading vectors. In this paper, we show how to recover the loading vectors from the auto-encoder weights.

"Loss Landscapes of Regularized Linear Autoencoders" by Daniel Kunin, Jonathan M. Bloom, Aleksandrina Goeva, Cotton Seed develops this idea further.

Autoencoders are a deep learning model for representation learning. When trained to minimize the Euclidean distance between the data and its reconstruction, linear autoencoders (LAEs) learn the subspace spanned by the top principal directions but cannot learn the principal directions themselves. In this paper, we prove that $L_2$-regularized LAEs learn the principal directions as the left singular vectors of the decoder, providing an extremely simple and scalable algorithm for rank-$k$ SVD. More generally, we consider LAEs with (i) no regularization, (ii) regularization of the composition of the encoder and decoder, and (iii) regularization of the encoder and decoder separately. We relate the minimum of (iii) to the MAP estimate of probabilistic PCA and show that for all critical points the encoder and decoder are transposes. Building on topological intuition, we smoothly parameterize the critical manifolds for all three losses via a novel unified framework and illustrate these results empirically. Overall, this work clarifies the relationship between autoencoders and Bayesian models and between regularization and orthogonality.