Following on from: What're the differences between PCA and autoencoder?
If I want to do dimensionality reduction and restrict myself to using a linear activation for my autoencoder, is there any reason to use an AE over Principal Component Analysis? PCA minimises the squared distance between our data restricted in some subspace and the true representation and has a closed form solution so is very fast - an AE really goes round the houses to achieve this and the solution will have no better loss i.e. the result of the loss function being optimised could be at best the same as for PCA.
So I conclude that I should only ever use an autoencoder for dimensionality reduction if I want to incorporate a non-linear activation function, or use multiple layers, or use a convolutional layer etc. Is this a fair conclusion?