In CNN architectures, can we say that the layer before the last softmax layer is compressed version of input image?

  • 1
    $\begingroup$ You should define compression $\endgroup$
    – kbrose
    Oct 22 '19 at 18:10

What is your definition of "compressed"?

  1. If the penultimate layer of your network has $d$ units, then the image is represented as a vector in $\mathbb{R}^d$. If $d$ is smaller than number of elements the original image, one could argue that it is "compressed" in this sense.

  2. On the other hand, the computer science usage of the word "compression" focuses on (lossy or lossless) reconstruction of the original input on the basis of a different representation (such as the $d$ elements of the hidden representation).

    a. There are neural networks which can do this, such as super-resolution networks or strategies.

    b. But other networks, such as classification networks, do not reproduce the original image; instead, they make a decision about what the image contains, so the meaning of the hidden representation is oriented around representing the inputs in terms of the target classes.

    c. Alternatively, even if the representation in $\mathbb{R}^d$ is not aimed at reproducing the original input, such as in a classification network, you could argue that this hidden representation is a "distillation" of the relevant information about class membership. While hidden representations clearly store information about an image (or any input generally), I don't believe they're easily interpreted. For some discussion, see: Can't deep learning models now be said to be interpretable? Are nodes features?

In each of these alternative perspectives, the core of the issue is what you mean when you say "compression," and implicitly depend on what problem you're trying to solve.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.