(I don't much about deep learning, but have been playing with a few things and have some questions.)

I took the (pretrained on imagenet) resnet18 model from pytorch, removed the last fully-connected layer and ran the STL-10 images (96 pixels square) through it. The result is a 512-dimensional data set. It seems that this data is spherical in that the variance in norms is $\approx0.01$ and all of the data is in a spherical shell of radius between 20 and 22. It also seems that none of the features are redundant as seen on this PCA explained variance curve:

enter image description here

I did something similar, running the CIFAR-10 images (32 pixels square) through the pretrained densenet121 model from pytorch (with the last linear layer removed), getting 1024-dimensional data. This data doesn't seem to be spherical at all, and there is a lot of redundancy among the features:

enter image description here

I assume the observations above come from the architecture of the models, which I know nothing about really. My questions are:

  1. Are there any rules of thumb about the distribution of image data run through some of the popular networks?
  2. Is there a reason why the resnet produces spherically distributed data?
  3. Is there a reason why the densenet produces a lot of dependent data?

(Note: To use pytorch models, images were resized to 256 pixels square and center-cropped at 224 pixels square.)

  • 1
    $\begingroup$ You should maybe compare the two curves on the same scale? $\endgroup$
    – Jon Nordby
    Feb 10, 2021 at 7:54
  • $\begingroup$ In (3) you say that you resized the images. Are you making small images larger? What size are the images before and after resizing? What method do you use for resizing? $\endgroup$
    – Sycorax
    Feb 10, 2021 at 16:44
  • $\begingroup$ If you want to compare the two models, I would recommend using the same data. $\endgroup$
    – Jon Nordby
    Feb 10, 2021 at 18:41


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.