3
$\begingroup$

I have a regression task where the label is varying from about 0.001 to 1000. One of the feature called group, for example, group A corresponding label from 0-0.1 and group G corresponding label from 500-700. Each group has about 500,000 rows.

I trained many models based on MSE as the objective function, but I notice that if I plot real vs predict value for each group, the plot in group A looks very bad because the predict value is 2 times larger that real value (e.g. 0.02 vs 0.04) and group G plot looks good (e.g. 650 vs 670)

Are there any way to solve this? I think I can scale my label or MAPE can be a better objective function for the model but I don't know how to do it. And I can't build different models for different groups

$\endgroup$
6
$\begingroup$

MAPE is defined as

$$ \mathcal{L} = \sum_{i=1}^n \frac{|y_i - \hat{y}_i|}{y_i} \tag{1} $$

Option 1: Weighted Least Squares

First, an approximate (but very convenient) solution. By using weighted least squares with sample weights of $w_i = 1/y_i$, we obtain the following loss function:

$$ \mathcal{L} = \sum_{i=1}^n w_i (y_i - \hat{y}_i)^2 \tag{2} $$

Loss (2) is still using MSE and therefore can be run with all the standard tools, but is a heck of a lot closer to (1) than unweighted least squares. In particular it will force the algorithm to fit very small y values like 0.02 much more tightly than large ones.

Option 2: Weighted Median Regression

By combing the weighting trick with a shift from MSE to MAE (Mean Absolute Error) we can get MAPE exactly. I saw this trick in this paper but once you see it it's really quite obvious in hindsight. Algorithms that minimize MAE are sometimes called median regression. They aren't as common as least squares optimizer (because the resulting linear programming optimization problem is more complicated) but you can still easily find off the shelf implementations for your language of choice. This gives (3) as our loss function, and with $w_i = 1/y_i$ this is exactly MAPE.

$$ \mathcal{L} = \sum_{i=1}^n w_i|y_i - \hat{y}_i| \tag{3} $$

This option gives you a MAPE loss function exactly, but this must be counterbalanced by the practical issues of switching to a median regression.

Option 3: Log Transform

A third option is to simply apply a log transform to y. If we use minimize MSE of $\log(y)$ our loss function is:

$$ \begin{align} \mathcal{L} & = \sum_{i=1}^n (\log(\hat{y}_i)-\log(y_i))^2 \\ & = \sum_{i=1}^n \Big( \log\frac{\hat{y}_i}{y_i} \Big)^2 \\ \end{align} \tag{4} $$

This has the property that it is the ratio $\hat{y}_i/y_i$ that matters, so that a prediction of 0.04 when the true value is 0.02 receives just as large a penalty as predicting a 1,300 instead of a 650. Thus, it would see 670 as quite on the money (with a small loss) and 0.04 instead of 0.02 as very far from the mark (with a huge loss) which is more or less what you want.

The log transform option is only available if your data is strictly positive and may have other undesirable properties such as asymmetric confidence intervals or ending up with residuals that don't appear to be normally distributed. On the other hand, a log transform sometimes helps with these issues, so apply standard model assumption checking and validation techniques. Here is a classic (highly-upvoted) question and answer which discusses when and why a log transform would be appropriate.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.