# Does it make sense to talk about the standard deviation of RMSE?

I want to plot the RMSE of some models to compare their performance in a dataset. I'd like also to include error bars, because I know they may have very different standard deviations.

The problem is, does it even make sense to talk about standard deviation of RMSE? I thought about taking the square root of the standard deviation of the MSE, but I don't know if that's what I need.

• You can talk about the variance of any statistic. It is turtles all the way down. – tchakravarty Nov 19 '17 at 5:39

RMSE is the square root of MSE. But the answer to your question depends on if you are talking about the MSE of a predictor versus an estimator.

# MSE of Estimator

MSE of an estimator is a fixed quantity, and has no variance. So it makes no sense to talk about the SD of the MSE.

Consider a special case with model, $M_1$, $Y_i = \beta$ for $n$ $iid$ observations. Suppose $E[Y_i]=\beta$ and $Var[Y_i] = \phi$, $\forall i$. An unbiased estimator is for $\beta$ is $\hat \beta = n^{-1} \sum_i Y_i$. Now $\hat \beta$ is a function of a random sample and so is random itself. If it's random, it has variance. Since it is unbiased, $MSE[\hat \beta]=Var[\hat\beta] = \phi$. So the MSE is constant.

$Var\big[MSE[\hat\beta]\big]=Var\big[\phi\big]=0$.

# MSE of a Predictor

This is a function of a random sample so it is itself random and therefore has variance. Consider predictor from the model above, $M_1$

$MSE_{pred} = n^{-1} \sum_i(Y_i - \hat Y)^2 = n^{-1} \sum_i(Y_i - \hat \beta)^2$

$\hat \beta$ is a function of data (random). $Y_i$ is the data (also random), so the whole MSE is a statistic - so is itself random. So it has variance and we can meaningfully talk about the SD of the MSE.

• Right, I can calculate the SD of the MSE, but if want it to be in the same unit as the RMSE, should I just take the square root? – erickrf Nov 19 '17 at 19:05
• In that case I'd take the square root first, then calculate the SD. That's because in general $Var(f(\hat \beta)) \neq f(Var(\hat \beta))$. In your case $f$ is the square root function. Does that address your question? – AOGSTA Nov 19 '17 at 19:51
• If I understood correctly, you're telling me to take the SD over the absolute error... is it right? – erickrf Nov 19 '17 at 19:59
• Not sure what you mean by absolute error. I think you want $SD(RMSE)$, but you only have $SD(MSE)$. You posit taking $\sqrt{SD(MSE)}$, but that is not what you need since $\sqrt{SD(MSE)} \neq SD(\sqrt{MSE})$ You need to calculate RMSE and then take the SD of this RMSE. You can't use the square root of the MSE's SD. – AOGSTA Nov 19 '17 at 20:11
• Right, I understand that. As absolute error, I meant that $\sqrt{MSE} = |Y_i - \hat{Y}|$ – erickrf Nov 19 '17 at 20:17