Suppose you have a network representing an autoencoder (AE). Let's assume it has 90 inputs/outputs. I want to batch-train it with batches of size 100. I will denote my input with
x and my output with
Now, I want to use the MSE to evaluate the performance of my training process. To my understanding, the input/output dimensions for my network are of size (100, 90).
The first part of the MSE calculation is performed element-wise, which is
(x - y)²
so I end up with an matrix of size (100, 90) again. For better understanding of my problem, I will arbitrarily draw a matrix of how this looks now:
[[x1 x2 x3 ... x90], # sample 1 of batch [x1 x2 x3 ... x90], # sample 2 of batch . . [x1 x2 x3 ... x90]] # sample 100 of batch
I have stumbled across various versions of calculating the error from now on. Goal of all versions is to reduce the matrix to a scalar, which can then be optimized.
Sum over the quadratic errors in the respective sample first, then calculate the mean of all samples, e.g.:
v1 = [ SUM_of_qerrors_1, # equals sum(x1 to x90) SUM_of_qerrors_2, ... SUM_of_qerrors_100 ] result = mean(v1)
Calculate mean of quadratic errors per sample, then calculate the mean over all samples, e.g.:
v2 = [ MEAN_of_qerrors_1, # equals mean(x1 to x90) MEAN_of_qerrors_2, ... MEAN_of_qerrors_100 ] result = mean(v2)
Personally, I think that version 1 is the correct way to do it, because the commonly used crossentropy is calculated in the same manner. But if I use version 1, it isn't really the MSE.
Can anybody clarify this, please?
In case of variational autoencoders, there is a second error term, that is added to the equation, called KL-divergence.
So after doing the first dimension reduction (here after version 1), the KL divergence vector of the same dimension is added, and then the second dimension reduction is performed:
v1 = [ SUM_of_qerrors_1, SUM_of_qerrors_2, ... SUM_of_qerrors_100 ]
v1 = v1 + KL_vector
result = mean(v1)
Does this make any difference in this case?