3
$\begingroup$

In a simple neural network, having more nodes on an input layer that on the next layer performs a compression or dimension reduction similar to what PCA does. The fewer nodes encode in a combination some kind of information that is in the previous layer.

While the forward computation is structurally similar to PCA , the weights form a matrix, it is not equivalent. That is, an autoencoder reduces dimensions as does PCA, but there is no gurantee of orthogonality or correspondence to eignevalues.

Is there an activation function and loss function for one layer (or larger more complicated architecture and choice of activation and loss functions and backprop alternative) that does converge to the PCA coefficients?

That is, is there some way to get a weight matrix that is orthogonal -and- the next level of nodes correspond to the eigensystem (sortable by eigenvalues)?

The motivation is to 'do everything' with a neural network architecture rather than use processes outside of the NN model. This way one could remove collinearity for modeling non-linear subspaces.

$\endgroup$
3
  • 2
    $\begingroup$ Just a comment but - I think you would be trying to get an approximate answer (via optimization in neural network) to a problem that already has an exact analytical solution. So I don't see anything that could be gained. $\endgroup$ Apr 7 at 19:47
  • 2
    $\begingroup$ See stats.stackexchange.com/a/120096/53084 $\endgroup$
    – rep_ho
    Apr 7 at 19:48
  • $\begingroup$ @KarolisKoncevičius Yes, there are alternatives outside of a NN that will do it faster (SVD). The gain is more theoretical, ie that it is possible but also not exorbitantly more expensive. $\endgroup$
    – Mitch
    Apr 7 at 21:28

1 Answer 1

3
$\begingroup$

Just doing PCA inside of a neural network is not much of a stretch, since the most naïve implementation will simply employ gradient updates to compute the $QR$ algorithm for the covariance matrix.

It's well-known that find a rank $k$ approximation to the data. And in the particular sense of minimizing certain metrics of reconstruction error, this approximation is optimal. However, there is no guarantee that the estimated matrices will be orthogonal, nor to components that maximize variance; indeed, we would expect the matrices to merely span the rank $k$ PCA solution simply because the generic auto-encoder optimization task does not impose these constraints.

But modern neural network libraries implement methods to introduce orthogonality constraints to matrices. For example, PyTorch does this with parameterizations, with a specific method for orthogonality constraints. We can likewise use parameterizations to do things like enforce that a weight matrix is triangular (for instance, by using a binary mask).

Finally, the $QR$ algorithm is a method to estimate the eigenvalues of a square matrix. PCA is a decomposition of the covariance matrix, which is square.

This is somewhat roundabout, and I don't believe implementing this homebrew PCA method is a good solution. I expect there to be superior methods to finding a nice low-rank representation of the data, even if the data are large.

Moreover, a further refinement would work on the data matrix directly, instead of the covariance matrix.

Incidentally, PyTorch also implements SVD (which can be used to do PCA: Relationship between SVD and PCA. How to use SVD to perform PCA?).


This leaves much to be desired; for instance, one of the main strengths of neural networks is that they can achieve state-of-the-art results by streaming batches of data, instead of requiring access to all of the data at once. And the sketch above assumes that you're forming the covariance matrix directly, instead of batches of raw data.

These papers outline several different methods to use neural networks to estimate PCA in more sophisticated ways. As I have time, I'll expand this answer to summarize the key points.

Fyfe, Colin. "A neural network for PCA and beyond." Neural Processing Letters 6 (1997): 33-41.

Migenda N, Möller R, Schenck W (2021) Adaptive dimensionality reduction for neural network-based online principal component analysis. PLoS ONE 16(3): e0248896. https://doi.org/10.1371/journal.pone.0248896

Du, Ke-Lin, and Madisetti NS Swamy. Neural networks and statistical learning. Springer Science & Business Media, 2013.

Kong, Xiangyu, Changhua Hu, and Zhansheng Duan. Principal component analysis networks and algorithms. Singapore: Springer Singapore, 2017.

Bartecki, K. (2012). Neural Network-Based PCA: An Application to Approximation of a Distributed Parameter System. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds) Artificial Intelligence and Soft Computing. ICAISC 2012. Lecture Notes in Computer Science(), vol 7267. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-29347-4_1

P. Pandey, A. Chakraborty and G. C. Nandi, "Efficient Neural Network Based Principal Component Analysis Algorithm," 2018 Conference on Information and Communication Technology (CICT), Jabalpur, India, 2018, pp. 1-5, doi: 10.1109/INFOCOMTECH.2018.8722348.

$\endgroup$
8
  • 1
    $\begingroup$ They will be orthogonal but they won't be ordered by variance explained as in PCA, so the weights and scores won't be there same. $\endgroup$
    – rep_ho
    Apr 7 at 20:07
  • 1
    $\begingroup$ Sure, but it will find the first $k$ principle components, which seems perfectly adequate! Ordering them is trivial. $\endgroup$
    – Sycorax
    Apr 7 at 20:30
  • 1
    $\begingroup$ I think that they won't be principal components, it will have a span of the first k principal components but the whole matrix will be rotated differently. $\endgroup$
    – rep_ho
    Apr 7 at 20:35
  • 2
    $\begingroup$ @rep_ho Ah, I see what you're saying. Yes, this is a flaw with the previous version. I think you can still get to PCA using parameterizations for (1) an orthogonal matrix $Q$ and (2) a triangular matrix $R$ because these two objects compose the $QR$ method to find eigenvectors and eigenvalues of a square matrix $X$, which is PCA when $X$ is a covariance matrix. The question then becomes whether this can be represented as a NN, which I think it can: minimize an appropriate norm of $Q^\top X - R$ because we seek $X = QR$. $\endgroup$
    – Sycorax
    Apr 7 at 21:19
  • 1
    $\begingroup$ @Mitch The documentation link I provided for orthogonal parameterizations mentions three alternative strategies for enforcing orthogonality. $\endgroup$
    – Sycorax
    Apr 7 at 21:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.