How to quantify the fluctuation of an error?

Question

First of all sorry for the bad title, but unfortunately, I can't think of a better one at the moment. Hopefully, that will change when my question is answered.

Let's say I have two sets of values: The target values $T_i$ and the actual values $X_i$ and I want to quantify the errors $E_i=X_i-T_i$ using a single number. Due to the fact that I don't want negative and positive to compensate each other and the fact that I'd like to punish outliers, I decided to use the Root Mean Square Error $\text{RMSE}$.

Now the question: How can I quantify the fluctuation of the errors in a way that can be compared with the $\text{RMSE}$? My first idea was to use the standard deviation. After all, the $\text{RMSE}$ is a mean value and it is a plausible assumption, that the errors $E_i$ are normally distributed. However, I think the standard deviation is "incompatible" with the $\text{RMSE}$, because unlike the $\text{RMSE}$ it does not operate on the squared errors.

What are your recommendations?

EDIT: I'll try to make things clearer. What I want to do is to quantify the error of the $\text{RMSE}$. After all it is a mean, so there must be a way to tell how much the values, this mean is based on, fluctuate. Let's say I would use the $\text{MAE}$ instead of the $\text{RMSE}$. Then I could easily quantify the fluctuation of the absolute errors using their standard deviation. However, to apply the same approach to the $\text{RMSE}$, I would have to calculate the square root of the standard deviation of the squared errors, which seems a little odd to me.

Answer 1

From what I can make out, you have two requirements:

1. Ensure that positive and negative errors don't cancel out. So, you square the errors.
2. You want to punish the models which have large errors for some predictions (but possibly lower errors elsewhere) more than models which have no large outlier errors(but possibly higher errors elsewhere). This is done by, again, squaring the errors. The square really amplifies the outliers. (For example, a model with errors 10 and 0 has squared-sum-of-errors of 100 whereas a model with errors 5 and 5 has squared-sum-of-errors of 50).

If you notice, both your requirements imply the use of squared errors. Whether or not you take the root after summing the squared-errors has no impact on meeting the above requirements.