Weighted Root Mean Square Error I have two models that predicted a value that is area based. I want to measure the RMSE to compare the models, but I would like it to be weighted for area so that errors on a large area are given more weight then errors on a small area. I have come up with the following function in R:
weighted.rmse <- function(actual, predicted, weight){
  sqrt(mean((predicted-actual)^2*weight/sum(weight)))
}

For weight I will use the area of prediction. So to my question, is this a valid method for model comparison? Is there a better way?
 A: This is a very old thread, but I would change David Dickson's function as follows.
weighted.rmse <- function(actual, predicted, weight){
    sqrt(sum((predicted-actual)^2*weight)/sum(weight))
}

Tim's answer is only valid if weights sum to 1, but this function generalizes it so that it is valid with any (non-normalized) set of weights.
A: As already noticed by whuber in a comment, it is not clear if your procedure of setting weights is valid. Notice that in non-weighted RMSE larger areas already have greater weight on the estimate since they are larger, so they appear more often in your data. That is why, as suggested, people rather down-weight such subpopulations, so that the final estimate treats all the subpopulations more evenly.
However if you wanted to use weighted RMSE, then recall that RMSE is by design pretty close to standard deviation, so why not look at how weighted variance is calculated?
$$ \sigma^2 = \sum_{i=1}^n w_i (x_i - \bar x)^2 $$
where weights are non-negative and $\sum_{i=1}^n w_i = 1$. The same you can take weighted RMSE as
$$ \text{RMSE} = \sqrt{\sum_{i=1}^n w_i (\hat x_i - x_i)^2} $$
Notice that we take sum of weighted differences, not the mean. Unweighted mean is the same as weighted mean with weights that are all equal to $w_i = 1/n$, so if you took arithmetic mean, it would be like dividing RMSE by $n$ second time.
Check also:
Weighted Variance, one more time
A: If you do not mind doing some reading, I recommend looking up Sampling: Design and Analysis by Lohr or Sampling by Thompson for examples on model based weighting schemes for mean squared error (MSE). I'm sure you'll find copies online by doing a simple Google search. Since your data seems deal with area (location), I recommend reviewing the chapters on Spatial Sampling in Sampling.
Note that you should try to understand how your data was sampled (obtained) as that will affect the weights.
