1
$\begingroup$

The mean squared error for two vectors is calculated using

$$mse(x, y) = \frac{1}{n}\sum_{i=1}^n(x_i-y_i)^2\text{, with }x, y \in \mathbb{R^n}\text{.}$$

That is straightforward, but I have a hard time figuring out the correct way to apply this formula to machine learning algorithms where you have to deal with multiple samples which would be stored in matrices such as $X, Y \in \mathbb{R^{n\times m}}$. In such a matrix there is a total of $m$ samples and each sample has $n$ features.

What would the correct definition of MSE be in this case? Do you divide by $m$ (the number of sample vectors)

$$mse(X, Y) = \frac{1}{m} \sum_{i=1}^m \sum_{j=1}^n (x_{ij} - y_{ij})^2\text{,}$$

or do you divide by $n \cdot m$ (the number of scalars in the matrix)

$$mse(X, Y) = \frac{1}{m\cdot n} \sum_{i=1}^m \sum_{j=1}^n (x_{ij} - y_{ij})^2\text{?}$$

I'm sorry if this is a silly question. It may very well be since statistics is not something I deal with regularly.

$\endgroup$
4
  • $\begingroup$ For vectors, each dimension usually means something and it wouldn't make sense to subtract the first dimension of one vector from the second dimension of a different vector (i.e. $x_1 - y_2$). In this case, is there anything wrong subtracting the first sample of one matrix from the second sample of a different matrix? (i.e. $x_{1j} - y_{2j}$) $\endgroup$ – Eric Farng May 16 '15 at 1:54
  • $\begingroup$ @EricFarng I need $mse$ to return a single number that I can use as the neural network's error. It needs to be a scalar and not a vector so I can then run the back-propagation algorithm. $\endgroup$ – Paul Manta May 16 '15 at 6:09
  • $\begingroup$ In that case, I believe the two are equivalent since the difference can be absorbed into the learning rate. $\endgroup$ – Eric Farng May 16 '15 at 12:15
  • $\begingroup$ It doesn't really matter if you add some constant to the objective function or multiply it by a positive number since optimum values doesn't change The common practice is division by number of samples ("m") as it gives average over all samples. $\endgroup$ – yasin.yazici May 16 '15 at 16:53
1
$\begingroup$

MSE is Mean Square Error. So you first of all must compute your error (get different between vectors), compute it square and for total result compute your mean. For example: $$ Y = \begin{bmatrix} 1 & 2\\ 3 & 4\\ 2 & 0 \end{bmatrix} $$

$$ \hat{Y} = \begin{bmatrix} 0.5 & 2.5\\ 1 & 1\\ 1 & 1 \end{bmatrix} $$

Where $Y$ is your target result and $\hat{Y}$ is your output. Every matrix contains 3 samples and every sample has 2 features. SO the MSE for this situation would be $$ MSE = \frac{1}{3 \cdot 2} \sum_{i=1}^3 \sum_{i=1}^2 (Y - \hat{Y}) ^ 2 = \frac{1}{6} \sum_{i=1}^3 \sum_{i=1}^2 \begin{bmatrix} 0.5 & -0.5\\ 2 & 3\\ 1 & -1 \end{bmatrix} ^ 2 = \frac{1}{6} \sum_{i=1}^3 \sum_{i=1}^2 \begin{bmatrix} 0.25 & 0.25\\ 4 & 9\\ 1 & 1 \end{bmatrix} = \frac{1}{6} \sum_{j=1}^3 \begin{bmatrix} 0.5\\ 13\\ 2 \end{bmatrix} = \frac{15.5}{6} \approx 2.583 $$

As for me, coefficient for which you divide total sum result didn't play a huge role. It exist to control your error result in normal bounds. If you didn't use it you will also get valid error result but for large datasets number would be really big.

If you try use this error in some neural networks you must understand that this fraction just a scalar which in training step would be multiply by step which is also a scalar. So in learning process this value control your step, but if you would remove this fraction and will decrease your step the learning result would be the same if you even didn't divide your total error sum by number of elements in matrix.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.