# Correct way to calculate MSE for autoencoders with batch-training

Suppose you have a network representing an autoencoder (AE). Let's assume it has 90 inputs/outputs. I want to batch-train it with batches of size 100. I will denote my input with x and my output with y.

Now, I want to use the MSE to evaluate the performance of my training process. To my understanding, the input/output dimensions for my network are of size (100, 90).

The first part of the MSE calculation is performed element-wise, which is

(x - y)²


so I end up with an matrix of size (100, 90) again. For better understanding of my problem, I will arbitrarily draw a matrix of how this looks now:

[[x1 x2 x3 ... x90],    # sample 1 of batch
[x1 x2 x3 ... x90],    # sample 2 of batch
.
.
[x1 x2 x3 ... x90]]    # sample 100 of batch


I have stumbled across various versions of calculating the error from now on. Goal of all versions is to reduce the matrix to a scalar, which can then be optimized.

Version 1:

Sum over the quadratic errors in the respective sample first, then calculate the mean of all samples, e.g.:

v1 =
[ SUM_of_qerrors_1,        # equals sum(x1 to x90)
SUM_of_qerrors_2,
...
SUM_of_qerrors_100 ]

result = mean(v1)


Version 2:

Calculate mean of quadratic errors per sample, then calculate the mean over all samples, e.g.:

v2 =
[ MEAN_of_qerrors_1,        # equals mean(x1 to x90)
MEAN_of_qerrors_2,
...
MEAN_of_qerrors_100 ]

result = mean(v2)


Personally, I think that version 1 is the correct way to do it, because the commonly used crossentropy is calculated in the same manner. But if I use version 1, it isn't really the MSE.

Thank you!

EDIT:

In case of variational autoencoders, there is a second error term, that is added to the equation, called KL-divergence.

So after doing the first dimension reduction (here after version 1), the KL divergence vector of the same dimension is added, and then the second dimension reduction is performed:

v1 = [ SUM_of_qerrors_1, SUM_of_qerrors_2, ... SUM_of_qerrors_100 ]

v1 = v1 + KL_vector

result = mean(v1)

Does this make any difference in this case?

Error in Version 1 is $$90$$ times (assuming your number of features is $$90$$ based on your example) of Error in Version 2. They're both minimized at the same parameter set. It's just the gradients will be $$90$$ times larger in the first one, and via an adjusted learning rate they should converge to the same parameter set. So, you don't need to choose one versus another I think. Mathematically,
$$E_1=\frac{1}{n}\sum_{i=1}^n\sum_{j=1}^f e_{ij}^2, \ \ \ \ E_2=\frac{1}{n}\sum_{i=1}^n\left(\frac{1}{f}\sum_{j=1}^f e_{ij}^2\right)=\frac{1}{f}E_1$$