I am dealing with a regression problem (my targets could potentially take values between -inf to +inf).
To optimise my model, I have two objectives:
1) Predictions should be close to the targets.
2) The sign of my prediction should match the sign of the target.
For 1) I can simply use the square (L2) loss on my loss function. However, I am unsure which extra term should I add to my loss function to account for 2).
To illustrate this: If my target is y = 1.0, my loss should be larger for a prediction y_hat = -1.0, than for a prediction y_hat = 3.0.
I am solving the optimisation problem using Gradient Descent. In some sense, my problem is a classification-regression hybrid; I had in mind to use something similar to a hinge loss: max(0, -y * y_hat). However, since the target values are not bounded (they could be anywhere between -inf to inf), predicting larger absolute values is penalised more strongly than small absolute values, yielding very poor results.