# Optimization of Linear Autoencoder with SGD

I'm interested in the Linear Autoencoder(LAE), and I knew that, at convergence point, the subspace LAE learns is the same as the subspace PCA learns up to linear transformations. Also, the loss function has saddle points, and its local minima become global minima. Here, the problem setting is as discussed in "Neural Networks and Principal Component Analysis: Learning from Examples Without Local Minima"(1989)(http://www.vision.jhu.edu/teaching/learning/deeplearning19/assets/Baldi_Hornik-89.pdf)

It seems that those theoretical facts were studied and derived in late 1980s and 1990s because of the computational constraints of those times, and I'm grateful I have those results. However, I'm also interested in its practical side. More concretely, I want to know about the convergence rate and the way that the LAE recovers the principal subspace (i.e. which principal direction is tend to be learned faster than the others) when using the usual SGD algorithm.

Do you know if there are any works related to that topic? Although I found several articles related to that, they focus on different neural networks, not on LAE.

I think you will find this paper helpful: D. Kunin et al., "Loss Landscapes of Regularized Linear Autoencoders (2019). The authors discuss the convergence of linear autoencoders to the principal subspace and how regularization enables the recovery of the actual principal components (not just the subspace).

You might also find this paper helpful: E. Plaut, From Principal Subspaces to Principal Components with Linear Autoencoders (2018). This paper gives an elaborate introduction to LAE and their relationship with PCA.

• @elliotp Thanks a lot! The first one is insightful.. Nov 6, 2020 at 2:29