4
$\begingroup$

Today I was in the middle of a particularly boring lecture. During this class I briefly let my mind wander. During this time I wondered:

Given a unique, but possibly similar, set of input data is it possible for the encoder to create overlapping codings for two or more different inputs?.

Suppose that I have an arbitrary set of inputs, $X$. In this input set, there are no duplicate data instances, but the data can be very simular. If you were to feed this data into a sufficiently trained autoencoder is it possible for the encoder to output the same codings for multiple different instances of the input set?

$\endgroup$
5
$\begingroup$

In some setups, not only they can, they need to. An idealized Denoising Autoencoder with a weak decoder would map any input+noise, as well as just input, to the same eventual latent code - its encoder would be just a lossless compression of the noiseless data, plus noise filters.

For a negative case, in a pathological scenario the latent encoding could collapse into a single vector, producing a single underfitted reconstruction with a local minimum of reconstruction cost.

That's just classical AEs. A VAE should produce overlapping codes, if you consider the code to be the sample rather than the distribution parameter, being N-dimensional bubbles in a compact and (approximately) continuous latent space.

$\endgroup$
2
$\begingroup$

There is more information going to the bottleneck than it is going out, so some input have to produce the same outputs.

$\endgroup$
  • 1
    $\begingroup$ Obviously, if you input all possible inputs (no matter how many), there are bound to be some duplicates in the latent space codings. But is there a way to tell how likely such "collisions" are? $\endgroup$ – jdw136 Jan 22 at 19:52
  • 1
    $\begingroup$ The information going into the auto-encoder may not be redundant. For example, if each input to the network is composed of repeated scalars for that example, only one scalar is needed. $\endgroup$ – Sycorax says Reinstate Monica Jan 22 at 19:52
  • 1
    $\begingroup$ I think you might be able to compute how much information are you loosing due to your bottleneck and from there there might be a way to compute likelihood of duplicates, but who knows. You should definitely be able to creating them yourself by inverting the network $\endgroup$ – rep_ho Jan 22 at 19:59
  • $\begingroup$ @SycoraxsaysReinstateMonica I liked your answer, you should have leave it up $\endgroup$ – rep_ho Jan 22 at 20:01
  • 1
    $\begingroup$ Hmm, isn't this effectively the same problem as trying to find the shortest program that can generate the input, subject to some extra constraints on the algorithm (i.e. the model architecture)? $\endgroup$ – jkm Jan 22 at 20:09
2
$\begingroup$

Trivially, if your bottleneck/representation layer uses ReLU activations and all of the inputs to that layer are less than 0, the encoding will be all 0s. So to produce such encodings, you'd just need to have two inputs that have the property that they get mapped "to the left side" of all the bottleneck layer ReLUs.

Or you have an auto-encoder with weights that are all 0 (maybe because you set the $L^2$ regularization too high and the model collapsed), this model assigns all inputs to the same code.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.