8
$\begingroup$

I have two models that predicted a value that is area based. I want to measure the RMSE to compare the models, but I would like it to be weighted for area so that errors on a large area are given more weight then errors on a small area. I have come up with the following function in R:

weighted.rmse <- function(actual, predicted, weight){
  sqrt(mean((predicted-actual)^2*weight/sum(weight)))
}

For weight I will use the area of prediction. So to my question, is this a valid method for model comparison? Is there a better way?

Improve this question
$\endgroup$
1
  • $\begingroup$ There are two considerations about which you could supply more information. (1) How exactly does the actual area influence the importance of the prediction? (2) How do the variances of the model errors change with actual area? Yours looks like an unusual circumstance because in most situations, (1) quantities associated with areas grow with the areas and (2) so do their variances. Moreover, often larger errors are more tolerable in larger areas. Both of those indicate one should downweight prediction errors as the area increases. $\endgroup$
    – whuber ♦
    Commented Aug 18, 2016 at 15:13

3 Answers 3

Reset to default
15
$\begingroup$

As already noticed by whuber in a comment, it is not clear if your procedure of setting weights is valid. Notice that in non-weighted RMSE larger areas already have greater weight on the estimate since they are larger, so they appear more often in your data. That is why, as suggested, people rather down-weight such subpopulations, so that the final estimate treats all the subpopulations more evenly.

However if you wanted to use weighted RMSE, then recall that RMSE is by design pretty close to standard deviation, so why not look at how weighted variance is calculated?

$$ \sigma^2 = \sum_{i=1}^n w_i (x_i - \bar x)^2 $$

where weights are non-negative and $\sum_{i=1}^n w_i = 1$. The same you can take weighted RMSE as

$$ \text{RMSE} = \sqrt{\sum_{i=1}^n w_i (\hat x_i - x_i)^2} $$

Notice that we take sum of weighted differences, not the mean. Unweighted mean is the same as weighted mean with weights that are all equal to $w_i = 1/n$, so if you took arithmetic mean, it would be like dividing RMSE by $n$ second time.

Check also:
Weighted Variance, one more time

Improve this answer
$\endgroup$
7
  • 1
    $\begingroup$ But that means the all weights should sum to 1 (i.e. normalized), am I right? So more generally, you should divide it with the (sum of weights) before sqrt-ing .. ? $\endgroup$
    – Hendy Irawan
    Commented Aug 14, 2018 at 11:15
  • 1
    $\begingroup$ @HendyIrawan why would it matter if you divided the loss by some constant, or not? It does not matter for minimizing the loss. $\endgroup$
    – Tim
    Commented Aug 14, 2018 at 11:47
  • $\begingroup$ I see. So the weights don't need to be normalized then? Which means the naming is not accurate, it should be "wRSSE" instead? Or... RWSSE (root weighted sum square error)? $\endgroup$
    – Hendy Irawan
    Commented Aug 14, 2018 at 13:02
  • 1
    $\begingroup$ @HendyIrawan if you want you can normalize it to have the consistent wRMSE name, but it doesn't matter. The weights need to be non-negative, that's all. $\endgroup$
    – Tim
    Commented Aug 14, 2018 at 13:08
  • $\begingroup$ I'm not mathematical enough to prove it, but I don't think unweighted RMSE is the same as this formula for weighted RMSE where every weight is 1/n. Perhaps you need to multiply the summation by n? $\endgroup$
    – Craig W
    Commented Nov 25 at 3:12
5
$\begingroup$

This is a very old thread, but I would change David Dickson's function as follows.

weighted.rmse <- function(actual, predicted, weight){
    sqrt(sum((predicted-actual)^2*weight)/sum(weight))
}

Tim's answer is only valid if weights sum to 1, but this function generalizes it so that it is valid with any (non-normalized) set of weights.

Improve this answer
$\endgroup$
1
  • $\begingroup$ My answer applies only to the case where weights sum to 1, they only need to be non-negative. If they don’t sum to 1, the metric changes by a constant, but if you use same weights to calculate the metric to compare different models it wouldn’t matter. There’s no reason why you would normalize the weights. $\endgroup$
    – Tim
    Commented Aug 18, 2021 at 18:13
1
$\begingroup$

If you do not mind doing some reading, I recommend looking up Sampling: Design and Analysis by Lohr or Sampling by Thompson for examples on model based weighting schemes for mean squared error (MSE). I'm sure you'll find copies online by doing a simple Google search. Since your data seems deal with area (location), I recommend reviewing the chapters on Spatial Sampling in Sampling.

Note that you should try to understand how your data was sampled (obtained) as that will affect the weights.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.