Just doing PCA inside of a neural network is not much of a stretch, since the most naïve implementation will simply employ gradient updates to compute the $QR$ algorithm for the covariance matrix.
It's well-known that autoencoders find a rank $k$ approximation to the data. And in the particular sense of minimizing certain metrics of reconstruction error, this approximation is optimal. However, there is no guarantee that the estimated matrices will be orthogonal, nor to components that maximize variance; indeed, we would expect the matrices to merely span the rank $k$ PCA solution simply because the generic auto-encoder optimization task does not impose these constraints.
But modern neural network libraries implement methods to introduce orthogonality constraints to matrices. For example, PyTorch does this with parameterizations, with a specific method for orthogonality constraints. We can likewise use parameterizations to do things like enforce that a weight matrix is triangular (for instance, by using a binary mask).
Finally, the $QR$ algorithm is a method to estimate the eigenvalues of a square matrix. PCA is a decomposition of the covariance matrix, which is square.
This is somewhat roundabout, and I don't believe implementing this homebrew PCA method is a good solution. I expect there to be superior methods to finding a nice low-rank representation of the data, even if the data are large.
Moreover, a further refinement would work on the data matrix directly, instead of the covariance matrix.
Incidentally, PyTorch also implements SVD (which can be used to do PCA: Relationship between SVD and PCA. How to use SVD to perform PCA?).
This leaves much to be desired; for instance, one of the main strengths of neural networks is that they can achieve state-of-the-art results by streaming batches of data, instead of requiring access to all of the data at once. And the sketch above assumes that you're forming the covariance matrix directly, instead of batches of raw data.
These papers outline several different methods to use neural networks to estimate PCA in more sophisticated ways. As I have time, I'll expand this answer to summarize the key points.
Fyfe, Colin. "A neural network for PCA and beyond." Neural Processing Letters 6 (1997): 33-41.
Migenda N, Möller R, Schenck W (2021) Adaptive dimensionality reduction for neural network-based online principal component analysis. PLoS ONE 16(3): e0248896. https://doi.org/10.1371/journal.pone.0248896
Du, Ke-Lin, and Madisetti NS Swamy. Neural networks and statistical learning. Springer Science & Business Media, 2013.
Kong, Xiangyu, Changhua Hu, and Zhansheng Duan. Principal component analysis networks and algorithms. Singapore: Springer Singapore, 2017.
Bartecki, K. (2012). Neural Network-Based PCA: An Application to Approximation of a Distributed Parameter System. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds) Artificial Intelligence and Soft Computing. ICAISC 2012. Lecture Notes in Computer Science(), vol 7267. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-29347-4_1
P. Pandey, A. Chakraborty and G. C. Nandi, "Efficient Neural Network Based Principal Component Analysis Algorithm," 2018 Conference on Information and Communication Technology (CICT), Jabalpur, India, 2018, pp. 1-5, doi: 10.1109/INFOCOMTECH.2018.8722348.