Let's assume that you have trained your (variational) autoencoder on MNIST digits. After some time, you check the result and decide that the reconstruction is pretty good. But this is highly subjective.
Imagine that you give the encoder a picture that show an 'X'. Looking at the output, the reconstruction should be really bad. But again, this is subjective.
I am looking for a way to objectively evaluate the performance for the current image. For VAE, the cost function during the training is a combination of a reconstruction error (MSE in my case) and the KL-divergence. Until now, I used the lone reconstruction error for this, as the dramatically rises when feeding an X.
The problem is: where do I draw a line between 'good' and 'bad' reconstructions and based on which objective measure can I do this?