Suppose you have a network representing an autoencoder (AE). Let's assume it has 90 inputs/outputs. I want to batch-train it with batches of size 100. I will denote my input with x and my output with y.

Now, I want to use the MSE to evaluate the performance of my training process. To my understanding, the input/output dimensions for my network are of size (100, 90).

The first part of the MSE calculation is performed element-wise, which is

(x - y)²

so I end up with an matrix of size (100, 90) again. For better understanding of my problem, I will arbitrarily draw a matrix of how this looks now:

[[x1 x2 x3 ... x90],    # sample 1 of batch
 [x1 x2 x3 ... x90],    # sample 2 of batch
 [x1 x2 x3 ... x90]]    # sample 100 of batch

I have stumbled across various versions of calculating the error from now on. Goal of all versions is to reduce the matrix to a scalar, which can then be optimized.

Version 1:

Sum over the quadratic errors in the respective sample first, then calculate the mean of all samples, e.g.:

v1 = 
[ SUM_of_qerrors_1,        # equals sum(x1 to x90)
  SUM_of_qerrors_100 ]

result = mean(v1)

Version 2:

Calculate mean of quadratic errors per sample, then calculate the mean over all samples, e.g.:

v2 = 
[ MEAN_of_qerrors_1,        # equals mean(x1 to x90)
  MEAN_of_qerrors_100 ]

result = mean(v2)

Personally, I think that version 1 is the correct way to do it, because the commonly used crossentropy is calculated in the same manner. But if I use version 1, it isn't really the MSE.

Can anybody clarify this, please?

Thank you!


In case of variational autoencoders, there is a second error term, that is added to the equation, called KL-divergence.

So after doing the first dimension reduction (here after version 1), the KL divergence vector of the same dimension is added, and then the second dimension reduction is performed:

v1 = [ SUM_of_qerrors_1, SUM_of_qerrors_2, ... SUM_of_qerrors_100 ]

v1 = v1 + KL_vector

result = mean(v1)

Does this make any difference in this case?


Error in Version 1 is $90$ times (assuming your number of features is $90$ based on your example) of Error in Version 2. They're both minimized at the same parameter set. It's just the gradients will be $90$ times larger in the first one, and via an adjusted learning rate they should converge to the same parameter set. So, you don't need to choose one versus another I think. Mathematically,

$$E_1=\frac{1}{n}\sum_{i=1}^n\sum_{j=1}^f e_{ij}^2, \ \ \ \ E_2=\frac{1}{n}\sum_{i=1}^n\left(\frac{1}{f}\sum_{j=1}^f e_{ij}^2\right)=\frac{1}{f}E_1$$

  • $\begingroup$ Thank you, that is true. Does this also apply in case the cost function has two parts, like it is the case with variational autoencoders? In this case, after the first dimension reduction, the KL-divergence column vector (same dimensions) is added, and then the mean is calculated. I suppose in this case, it would make a difference, right? Also, I have edited my original post for clarification. $\endgroup$ – DocDriven May 1 '19 at 13:12
  • 1
    $\begingroup$ @DocDriven it does make difference even if you add constant term, not only KL divergence. I've seen that the sum is being used, but there are implementations with unequal weights for KL and MSE terms. Here, there is another post in this forum considering the weighting problem: stats.stackexchange.com/questions/332179/… $\endgroup$ – gunes May 1 '19 at 16:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.