Ideas for a loss function to use for a cost sensitive problem setting?

Question

I have trained a model to perform regression on a dataset with MSE as its loss function. The y_real values are between 0 and 1.5 and MSE of test set is around 0.009 which if fine.

However, the regression is not really the goal and MSE is not very useful. Let me explain: My goal is to predict the y values that are over 1 as accurate as possible. So if y_real is over 1, I would like the model to predict it to be over 1 or if not I would like it to be as close to 1 as possible.

For values below 1, the cost should not be as high, and the cost should be lower the further we get from 1. For example, the cost of mispredicting y_real=0.3 and y_pred=0.1 should be much lower than y_real=1.1 and y_pred=0.9.

This seems like an imbalanced classification combined with cost function dependent on the distance of classes, which I can't wrap my head around.

Answer 1

You can continue to use MSE as a loss function but implement a weighted loss through weighted least squares. Each observation gets a weight proportional to its importance in the "true" loss function. For example, if we have a relative loss of $1$ for predicting values when $y_i < 1$ and of $10$ when $y_i \geq 1$, we would do the following:

x <- rnorm(100)
y <- 1 + x + rnorm(100)

# basic model performance 
e <- residuals(lm(y~x))
c(mean(e[y<1]^2), mean(e[y>=1]^2))

# weighted model performance
wts <- rep(1, length(x))
wts[y>=1] <- 10

e_wtd <- residuals(lm(y~x, weights=wts))

# Print out MSEs, y<1 then y>=1
c(mean(e[y<1]^2), mean(e[y>=1]^2))
c(mean(e_wtd[y<1]^2), mean(e_wtd[y>=1]^2))

with results:

> c(mean(e[y<1]^2), mean(e[y>=1]^2))
[1] 1.0568586 0.7156736

> c(mean(e_wtd[y<1]^2), mean(e_wtd[y>=1]^2))
[1] 2.082143 0.381666

The models are different, and, as we will see, the basic MSE is indeed worse for the weighted model, as we know it should be:

> summary(lm(y~x))

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)  1.04027    0.09516  10.932  < 2e-16 ***
x            0.90207    0.09788   9.216 6.09e-15 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.9455 on 98 degrees of freedom
Multiple R-squared:  0.4643,    Adjusted R-squared:  0.4588 

> summary(lm(y~x, weights=wts))

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)  1.54182    0.07735  19.934  < 2e-16 ***
x            0.71582    0.09092   7.873 4.74e-12 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 1.75 on 98 degrees of freedom
Multiple R-squared:  0.3874,    Adjusted R-squared:  0.3812