While training (Variational)-Autoencoder networks, I came along the paper by Higgins et al. "DARLA" where she stated:

The shortcomings of calculating the log-likelihood term [...] on a per-pixel basis are known and have been addressed in the past by calculating the reconstruction cost in an abstract, high-level feature space given by another neural network model, such as a GAN [..] or a pre-trained AlexNet [...]

Since there is no statement regarding why it is better to use feature-space instead of pixel-space, I am wondering why the results become better in the first case. I assume that since the per-pixel loss encourages the whole image to be trained correctly, a feature based loss can only fetch the perceptual similarity. Therefore, the per-pxel loss should always result in better outputs, or is there any important influence by the regularizer which I am missing?


Pixel space is fine for exact reconstructions, but suppose your decoder outputs the right answer, shifted over by a couple pixels. This will have a terrible loss in pixel space, but might be fine in feature space. This can result in the network being too conservative in some cases. For many applications, this makes "perceptual" losses superior, when exact reconstruction matters less than perceptual quality.

It also implicitly confers a Gaussian noise assumption, resulting in blurry outputs (as in VAEs) in pixel space, whereas "blurriness" in feature space may well still be nice and sharp in pixel space (depending on the manifold defined by the feature space of course).

| cite | improve this answer | |
  • $\begingroup$ But do perceptual losses still optimize the evidence lower bound, or is the introduction of these kinds of reconstructions a heuristic approach? There are nice approaches like VAEGAN or other publications by Sonderby which catch your answer, but do you have some good readings for your statement? $\endgroup$ – Tik0 Apr 4 '19 at 6:11
  • 1
    $\begingroup$ @Tik0 Well... it still optimizes an ELBO, although not the same one. The generative likelihood model $\log p(x|z)$ has simply changed. For references, I don't know of a specific one. Maybe Dosovitskiy and Brox Generating Images with Perceptual Similarity Metrics based on Deep Networks is relevant. $\endgroup$ – user3658307 Apr 4 '19 at 12:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.