Correct way to calculate MSE for autoencoders with batch-training

Question

Suppose you have a network representing an autoencoder (AE). Let's assume it has 90 inputs/outputs. I want to batch-train it with batches of size 100. I will denote my input with x and my output with y.

Now, I want to use the MSE to evaluate the performance of my training process. To my understanding, the input/output dimensions for my network are of size (100, 90).

The first part of the MSE calculation is performed element-wise, which is

(x - y)²

so I end up with an matrix of size (100, 90) again. For better understanding of my problem, I will arbitrarily draw a matrix of how this looks now:

[[x1 x2 x3 ... x90],    # sample 1 of batch
 [x1 x2 x3 ... x90],    # sample 2 of batch
 .
 .
 [x1 x2 x3 ... x90]]    # sample 100 of batch

I have stumbled across various versions of calculating the error from now on. Goal of all versions is to reduce the matrix to a scalar, which can then be optimized.

Version 1:

Sum over the quadratic errors in the respective sample first, then calculate the mean of all samples, e.g.:

v1 = 
[ SUM_of_qerrors_1,        # equals sum(x1 to x90)
  SUM_of_qerrors_2,
  ...
  SUM_of_qerrors_100 ]

result = mean(v1)

Version 2:

Calculate mean of quadratic errors per sample, then calculate the mean over all samples, e.g.:

v2 = 
[ MEAN_of_qerrors_1,        # equals mean(x1 to x90)
  MEAN_of_qerrors_2,
  ...
  MEAN_of_qerrors_100 ]

result = mean(v2)

Personally, I think that version 1 is the correct way to do it, because the commonly used crossentropy is calculated in the same manner. But if I use version 1, it isn't really the MSE.

Can anybody clarify this, please?

Thank you!

EDIT:

In case of variational autoencoders, there is a second error term, that is added to the equation, called KL-divergence.

So after doing the first dimension reduction (here after version 1), the KL divergence vector of the same dimension is added, and then the second dimension reduction is performed:

v1 = [ SUM_of_qerrors_1, SUM_of_qerrors_2, ... SUM_of_qerrors_100 ]

v1 = v1 + KL_vector

result = mean(v1)

Does this make any difference in this case?

Answer 1

Error in Version 1 is $90$ times (assuming your number of features is $90$ based on your example) of Error in Version 2. They're both minimized at the same parameter set. It's just the gradients will be $90$ times larger in the first one, and via an adjusted learning rate they should converge to the same parameter set. So, you don't need to choose one versus another I think. Mathematically,

$$E_1=\frac{1}{n}\sum_{i=1}^n\sum_{j=1}^f e_{ij}^2, \ \ \ \ E_2=\frac{1}{n}\sum_{i=1}^n\left(\frac{1}{f}\sum_{j=1}^f e_{ij}^2\right)=\frac{1}{f}E_1$$