1
$\begingroup$

I am experimenting with autoencoders for a very specific application, but cannot unfortunately go into the specific details of what I am doing yet (fingers crossed I can do so after I make some progress).

Traditionally autoencoders have the encoder followed by the decoder. After the encoder, I add both a normalization and Gaussian noise step before adding the decoder as shown below. However, instead of adding Gaussian noise like in the example code, I want to add "non-random noise" for each sample. By "non-random noise", I mean augmenting the encoder with a vector of values. So for each data sample, I want to add this precomputed vector of values for that sample (the vector will differ by sample, so it's dependent on the sample itself) at the encoder level as noise.

I am not super familiar with deep learning frameworks, but is there any way to do this in keras and if so, how? If not, can I do it using tensor flow instead? I have included some of the code I have gotten started with:

def build_autoencoder(input_dim, encoded_dim, noise_std):
  autoencoder = Sequential()

  # Encoder Layers
  autoencoder.add(Dense(encoded_dim, input_shape=(input_dim,), activation='relu'))
  autoencoder.add(Dense(encoded_dim, activation='linear'))

  # add normalization layer
  autoencoder.add(Lambda(lambda x: K.l2_normalize(x,axis=1)))

  # add Gaussian noise
  # TODO: replace this with non-random noise for each sample
  autoencoder.add(GaussianNoise(noise_std))

  # Decoder layer
  autoencoder.add(Dense(input_dim, activation='sigmoid'))

  return autoencoder
$\endgroup$
  • 1
    $\begingroup$ What exactly do you mean by "non-random precomputed noise"? It sounds just like another parameter... $\endgroup$ – Tim Feb 19 at 22:38
  • $\begingroup$ Hi! Good question. I reworded my explanation above, but let me know if it still doesn't make sense. Essentially, when I mean precomputed noise, I mean non-random, so not generated by some distribution like the Gaussian distribution, and each sample has its own vector of "noise" (vector of values to be added after the encoder). $\endgroup$ – Jane Sully Feb 20 at 5:08
2
$\begingroup$

Autoencoders take some data $\mathbf{x}$ and encode it into latent vector $\mathbf{z}$ of smaller dimensionality, to be able to reproduce $\mathbf{x}$ using the decoder:

$$ \mathbf{x} \to \mathsf{encoder}(\mathbf{x}) \to \mathbf{z} \to \mathsf{decoder}(\mathbf{z}) \to \mathbf{x}' $$

where we train it by minimizing the loss, that measures how different is the reproduced $\mathbf{x}'$ from $\mathbf{x}$.

Variational autoencoders treat $\mathbf{z}$ as a random variable, so that we can sample from its distribution and generate the possible outputs of $\mathbf{x}$. Treating $\mathbf{z}$ as a random variable may additionally make the model more flexible, and will enable us to quantify the uncertainties about the estimates, but those are not the main reasons why people use VAEs. This is done by making the encoder to learn the parameters ($\boldsymbol{\mu}, \boldsymbol{\sigma}$) of the distribution of $\mathbf{z}$, rather then learning $\mathbf{z}$ directly:

$$ \mathbf{x} \to \mathsf{encoder}(\mathbf{x}) \to\; \mathbf{z} \sim \mathcal{N}(\boldsymbol{\mu}, \boldsymbol{\sigma})\; \to \mathsf{decoder}(\mathbf{z}) \to \mathbf{x}' $$

"Adding the random noise" is just a computational trick to achieving this.

If you drop the "random noise" from variational autoencoder, it becomes a (non-variational) autoencoder, just an "autoencoder". Same, if the noise would be deterministic, because in such case $\mathbf{z}$ wouldn't be a random variable any more, so you wouldn't be able to sample from it, and the main feature of the variational autoencoder would be lost.

Moreover, if you want to add "precomputed vector of values for that sample (the vector will differ by sample, so it's dependent on the sample itself)", then what you say is something like:

$$ \mathbf{x} \to \mathsf{encoder}(\mathbf{x}) + f(\mathbf{x}) \to \mathbf{z} \to \mathsf{decoder}(\mathbf{z}) \to \mathbf{x}' $$

where $f(\mathbf{x})$ is some function of $\mathbf{x}$. Notice that this is just an autoencoder, that uses two encoder functions (sub-networks) and sums them, nothing more. Probably the same could be achieved with having single encoder, but with more parameters (neural networks are universal function approximators). If you mean that $f$ is going to have different structure then the $\mathsf{encoder}$, then this may make sense, but you would be using just an ensemble of different encoders as the encoder, where such approaches usually work better then the individual models.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks for going more in-depth into the conceptual aspects. This was definitely helpful. I was wondering if you could point me in the right direction in terms of implementation as well, using what I have above as a starting point/ Based off your explanation, it seems like I would need a way to specify a deterministic f(x), which in this case is just a vector of values for each training sample. Do I treat this as a separate network and concatenate? Not sure if the code shared in the question is a good starting point, but how would I modify that to achieve what you described? $\endgroup$ – Jane Sully Feb 25 at 5:45
  • $\begingroup$ @JaneSully in the linked blog post you can find nice introduction & examples. $\endgroup$ – Tim Feb 25 at 5:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.