Data reduction is often used to avoid overfitting and to enhance explainability. Popular data reduction techniques, such as SVD or PCA map/project high-dimensional data to a lower-dimensional representational space, where the dimensions are mutually orthogonal. Autoencoders (generally) also construct lower-dimensional representations of the input, but there is no guarantee of orthogonality. Are there any situations where applying SVD or PCA to have the assurance of orthogonality is not just the best thing to be doing?
1
-
3$\begingroup$ Orthogonality would be irrelevant to any nonlinear method, indicating there are likely situations where linear changes like SVD or PCA would indeed be counterproductive. $\endgroup$– whuber ♦Commented Apr 19 at 15:56
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
You could take any layer of an autoencoder, apply SVD to the inputs and you would just need to apply have a linear transformation to how those inputs were used and you end up with an identical solution.
So applying SVD to a layer is always going to be one of the best solutions...but just one of an infinite number of equally "good" solutions (if good is measured in reconstruction error), a set which is almost surely not orthogonal. So orthognalness may be orthogonal to whether it's good.
I think the utility of SVD in this case will likely come from whether the low-dimensional orthogonal space is more useful to an analyst. For example, we if apply SVD to the reduction space and now we see that each dimension seems to represent more distinct aspects of our data or provides for more meaningful clustering, that can be helpful. But it's not guaranteed by any means, we just have a hunch that it may help.
