1
$\begingroup$

Suppose I fit a linear regression $y = \beta x + \rm error$. In this situation, $x > 0$, $\alpha > 0$, and therefore $y > 0$. Moreover, the $\rm error$ is normally distributed with mean $0$ and standard deviation $\sigma$.

Now, if I calculate another error metric, percent error, which is $100*\rm predicted / actual$, then the error decreases with the value of $y$. That is, the model looks better with increasing values of $y$. This seem paradoxical. Any thoughts?

An example is suppose we are predicting sales as a function of amount of discount. Sales increases with the amount of discount. The model, on percent error basis, appears to perform well for higher amount of discount (higher sales). Any insights are appreciated.

$\endgroup$
  • 1
    $\begingroup$ What's alpha? Your regression doesn't contain one. $\endgroup$ – Glen_b Apr 21 '14 at 19:00
1
$\begingroup$

The model itself isn't any better or worse. The percent error statistic you are using assumes the distance of your predicted value from the observed value in units of how far the observed value is from $0$ is meaningful. If that is true, then situations where the data have the same residual SD, but the observed values are further from $0$ are more fortuitous situations. If that isn't true, then this statistic isn't appropriate.

The percent error metric probably makes more sense for a model in which the effects are multiplicative and you have a constant coefficient of variation, rather than constant variance. Another possible example might be a case where you estimate a certain amount of return on investment. Consider two cases: in the first you might make $\$1\pm\$2$ and in the second you might make $\$10\pm\$2$; in the first case you can loose money, but in the second case you make a decent return either way.

If you want to compare two models to the same dataset using this metric, it shouldn't make any difference, because the actual values will be the same in either case, so both would be standardized identically. What would happen is if the models differ in accuracy in different parts of the response space, that would be differentially weighted. For example, if one model yields predictions that are closer to the observed values when the observed values are high, and further when they are low, relative to the other model, the first would look better. Again, this assumes that metric is appropriate.

In addition, be aware that predicted values that are below the observed values will scale differently from those that are above. You may want to take the $\log$ of each of the percent errors to make that symmetrical and easier to interpret.

$\endgroup$
0
$\begingroup$

When you use OLS for the model specified, you minimize $\sum_i error_i^2$. So you force the regression to find the best fit which will have the behaviour you described, because your objective function does not include $y_i$ in it directly.

Instead you could minimize the relative errors $\sum_i (error_i/y_i)^2$, then the relative errors would be more uniformly minimized across different $y$ values. You could also apply the log transform to $y$ variable: $\ln y=\beta x + e$, in this case you'll also minimize the squared relative errors

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.