(I don't much about deep learning, but have been playing with a few things and have some questions.)
I took the (pretrained on imagenet) resnet18 model from pytorch, removed the last fully-connected layer and ran the STL-10 images (96 pixels square) through it. The result is a 512-dimensional data set. It seems that this data is spherical in that the variance in norms is $\approx0.01$ and all of the data is in a spherical shell of radius between 20 and 22. It also seems that none of the features are redundant as seen on this PCA explained variance curve:
I did something similar, running the CIFAR-10 images (32 pixels square) through the pretrained densenet121 model from pytorch (with the last linear layer removed), getting 1024-dimensional data. This data doesn't seem to be spherical at all, and there is a lot of redundancy among the features:
I assume the observations above come from the architecture of the models, which I know nothing about really. My questions are:
- Are there any rules of thumb about the distribution of image data run through some of the popular networks?
- Is there a reason why the resnet produces spherically distributed data?
- Is there a reason why the densenet produces a lot of dependent data?
(Note: To use pytorch models, images were resized to 256 pixels square and center-cropped at 224 pixels square.)