I'm newbie attempting the Zillow prize competition on Kaggle. I've read about the reason for using a relative error from Zillow's answer in their FAQ. (I don't think it's a Zillow-specific question but rather a general question in machine learning)
Q: Why did Zillow pick the log error instead of an absolute error metric such as RMSE?
A : Home sale prices have a right skewed distribution and are also strongly heteroscedastic, so we need to use a relative error metric instead of an absolute metric to ensure valuation models are not biased towards expensive homes. A relative error metric like the percentage error or log ratio error avoids these problems. While we report Zestimate errors in terms of percentages on Zillow.com because we believe that to be a more intuitive metric for consumers, we do not advocate using percentage error to evaluate models in Zillow Prize, as it may lead to biased models The log error is free of this bias problem and when using the natural logarithm, errors close to 1 approximate percentage errors quite closely. See this paper for more on relative errors and why log error should be used instead of percentage error.
I asked about this FAQ explanation here. So I mostly understand what "right skewed heteroscedastic distribution" means. But I still don't understand why these things are problems.
Here are the questions.
What does "biased towards expensive homes" mean? Does it mean that models have higher prediction error for expensive homes than cheap homes?
Why do we need to avoid making "biased" models? It's hard to say that models with higher error on expensive homes are bad because there are more cheap houses than expensive homes. I can't tell which is better.
What causes a model to be "biased towards expensive homes" and why does "bias" happen? Is it caused by "right skew" or "heteroscedasticity" or both?
EDIT after jon_simon's answer :
I'll explain my understanding and confusing point. Please correct me!
I understand that absolute error can be disproportionately influenced by the magnitude of Y and that's the reason we should use relative error metric. But what point I don't understand yet is "This problem can be caused by right-skew". (in jon_simon's answer)
There are more high-priced homes than low-priced homes and high-priced homes are more influence on absolute error metric like RMSE. So, if we use RMSE, the key to get better model is reduce prediction error for expensive homes more. Prediction error for low-priced homes will be relatively less important. So final chosen model should have ability of making lower error in high-priced model and I think this is the meaning of "biased" in my first question.
It is same that each high-priced home is more influence on RMSE than low-priced one. But the difference is the number of low-priced homes. There are much more low-priced homes than high-priced, and in terms of total error, low-priced homes get bigger influence than left-skewed distribution.
So, I think we don't know whether the final model will be biased towards low-priced homes or high-priced homes.(in right-skewed distribution)
In other words, this is a problem of this sort : "Is sum(low-priced homes' absolute error) bigger(or smaller) than sum(high-priced homes' absolute error)?"
I'm confused if we can say that right-skewed distribution surely causes 'biased towards expensive home' model.(with RMSE)