Can we say that last but one layer (that is fully connected layer) in CNN architectures is 'compressed' version of input image?

Question

In CNN architectures, can we say that the layer before the last softmax layer is compressed version of input image?

Answer 1

What is your definition of "compressed"?

1. If the penultimate layer of your network has $d$ units, then the image is represented as a vector in $\mathbb{R}^d$. If $d$ is smaller than number of elements the original image, one could argue that it is "compressed" in this sense.

2. On the other hand, the computer science usage of the word "compression" focuses on (lossy or lossless) reconstruction of the original input on the basis of a different representation (such as the $d$ elements of the hidden representation).

   a. There are neural networks which can do this, such as super-resolution networks or autoencoder strategies.

   b. But other networks, such as classification networks, do not reproduce the original image; instead, they make a decision about what the image contains, so the meaning of the hidden representation is oriented around representing the inputs in terms of the target classes.

   c. Alternatively, even if the representation in $\mathbb{R}^d$ is not aimed at reproducing the original input, such as in a classification network, you could argue that this hidden representation is a "distillation" of the relevant information about class membership. While hidden representations clearly store information about an image (or any input generally), I don't believe they're easily interpreted. For some discussion, see: Can't deep learning models now be said to be interpretable? Are nodes features?

In each of these alternative perspectives, the core of the issue is what you mean when you say "compression," and implicitly depend on what problem you're trying to solve.