I am trying to train an autoencoder with PyTorch on 2D images containing 2D Gaussian densities such as this:

Gaussian 2d

The images are of size 100x100 (I feed them into the autoencoder as 1x10000 tensors). The training data consists of densities with random locations inside the grid and varying standard deviations.

I get good results with my current architecture for such images (nearly identical outputs). But when I try out densities with very small standard deviation, the autoencoder has problems with the reconstruction. Here is an example input image:

Sparse input

Using the same architecture as above and training only on those sparse images, I get results like this:

Reconstructed sparse image

The location is reconstructed well but not the shape (e.g. no peak in the middle).

And here is the evolution of the loss during training:

Loss evolution

According to this thread: Autoencoder for sparse data it shouldn't be a problem that now the input is very sparse (most elements/pixels are zero). I already tried out different learning rates, batch_sizes and architectures but it didn't help.

My current architecture is a fully-connected autoencoder with hidden layer sizes as follows:

10.000 -> 1.024 -> 512 -> 256 -> 64 -> 256 -> 512 -> 1.024 -> 10.000

and ReLu activations in between. I am using MSE as the loss function.

Any ideas what could go wrong here?


1 Answer 1


I guess your model is overfitting (validation performance is worse than training) and it's probably because of one/all of the following:

  • The huge number of parameters (dense architecture) with -I assume- no regularization;
  • The distribution/diversity of your data (your gaussian distributions are all centered?);
  • The loss metric MSE is averaging over the number of pixels (10.000) even though most pixels are zeroed;

I would suggest starting from the bottom and using an asymmetric loss where the cost of predicting zero when it should be non-zero is different from the cost of predicting non-zero when it should be zero or using a weighted loss where you give more weight to the error on non-zero pixels.

  • $\begingroup$ Thank you for your suggestions! Concerning your second bullet point: My training data has random locations inside the 100x100 grid, I only chose those as examples. I also use some variaton for the value of standard deviation. You are also right that I use no regularization in the model. However, this is not a problem for images that have bigger standard deviations so I am wondering if this should be problem in this case? $\endgroup$
    – user149206
    Commented Jan 29, 2022 at 21:01
  • $\begingroup$ Not the person you're responding to but I think I had the same question and you didn't fully answer it yet: are the Gaussians always centered on a pixel, i.e. 100x100 possible discrete locations? Or are the Gaussian locations sampled from a continuous range, meaning that any particular Gaussian could have its peak not exactly at the center of a pixel, but in between pixel centers? In the latter case, the discretized values of the Gaussian would end up looking different each time, and that variability would be larger for narrower distributions, making them harder to learn & generalize. $\endgroup$ Commented Jan 29, 2022 at 21:39
  • $\begingroup$ The center is not necessarily on a single pixel because I randomly sample in the continuous space [0,100] x [0,100] for that. I will try out sampling only discrete values and give it a try! $\endgroup$
    – user149206
    Commented Jan 30, 2022 at 19:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.