PCA via a Neural Network

Question

In a simple neural network, having more nodes on an input layer that on the next layer performs a compression or dimension reduction similar to what PCA does. The fewer nodes encode in a combination some kind of information that is in the previous layer.

While the forward computation is structurally similar to PCA , the weights form a matrix, it is not equivalent. That is, an autoencoder reduces dimensions as does PCA, but there is no gurantee of orthogonality or correspondence to eignevalues.

Is there an activation function and loss function for one layer (or larger more complicated architecture and choice of activation and loss functions and backprop alternative) that does converge to the PCA coefficients?

That is, is there some way to get a weight matrix that is orthogonal -and- the next level of nodes correspond to the eigensystem (sortable by eigenvalues)?

The motivation is to 'do everything' with a neural network architecture rather than use processes outside of the NN model. This way one could remove collinearity for modeling non-linear subspaces.

Answer 1

Just doing PCA inside of a neural network is not much of a stretch, since the most naïve implementation will simply employ gradient updates to compute the $QR$ algorithm for the covariance matrix.

It's well-known that autoencoders find a rank $k$ approximation to the data. And in the particular sense of minimizing certain metrics of reconstruction error, this approximation is optimal. However, there is no guarantee that the estimated matrices will be orthogonal, nor to components that maximize variance; indeed, we would expect the matrices to merely span the rank $k$ PCA solution simply because the generic auto-encoder optimization task does not impose these constraints.

• What're the differences between PCA and autoencoder?
• Evaluating an autoencoder: possible approaches?

But modern neural network libraries implement methods to introduce orthogonality constraints to matrices. For example, PyTorch does this with parameterizations, with a specific method for orthogonality constraints. We can likewise use parameterizations to do things like enforce that a weight matrix is triangular (for instance, by using a binary mask).

Finally, the $QR$ algorithm is a method to estimate the eigenvalues of a square matrix. PCA is a decomposition of the covariance matrix, which is square.

This is somewhat roundabout, and I don't believe implementing this homebrew PCA method is a good solution. I expect there to be superior methods to finding a nice low-rank representation of the data, even if the data are large.

Moreover, a further refinement would work on the data matrix directly, instead of the covariance matrix.

Incidentally, PyTorch also implements SVD (which can be used to do PCA: Relationship between SVD and PCA. How to use SVD to perform PCA?).

---

This leaves much to be desired; for instance, one of the main strengths of neural networks is that they can achieve state-of-the-art results by streaming batches of data, instead of requiring access to all of the data at once. And the sketch above assumes that you're forming the covariance matrix directly, instead of batches of raw data.

These papers outline several different methods to use neural networks to estimate PCA in more sophisticated ways. As I have time, I'll expand this answer to summarize the key points.

Fyfe, Colin. "A neural network for PCA and beyond." Neural Processing Letters 6 (1997): 33-41.

Migenda N, Möller R, Schenck W (2021) Adaptive dimensionality reduction for neural network-based online principal component analysis. PLoS ONE 16(3): e0248896. https://doi.org/10.1371/journal.pone.0248896

Du, Ke-Lin, and Madisetti NS Swamy. Neural networks and statistical learning. Springer Science & Business Media, 2013.

Kong, Xiangyu, Changhua Hu, and Zhansheng Duan. Principal component analysis networks and algorithms. Singapore: Springer Singapore, 2017.

Bartecki, K. (2012). Neural Network-Based PCA: An Application to Approximation of a Distributed Parameter System. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds) Artificial Intelligence and Soft Computing. ICAISC 2012. Lecture Notes in Computer Science(), vol 7267. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-29347-4_1

P. Pandey, A. Chakraborty and G. C. Nandi, "Efficient Neural Network Based Principal Component Analysis Algorithm," 2018 Conference on Information and Communication Technology (CICT), Jabalpur, India, 2018, pp. 1-5, doi: 10.1109/INFOCOMTECH.2018.8722348.