I am dealing with a regression problem (my targets could potentially take values between -inf to +inf).

To optimise my model, I have two objectives:
1) Predictions should be close to the targets.
2) The sign of my prediction should match the sign of the target.

For 1) I can simply use the square (L2) loss on my loss function. However, I am unsure which extra term should I add to my loss function to account for 2).

To illustrate this: If my target is y = 1.0, my loss should be larger for a prediction y_hat = -1.0, than for a prediction y_hat = 3.0.
I am solving the optimisation problem using Gradient Descent. In some sense, my problem is a classification-regression hybrid; I had in mind to use something similar to a hinge loss: max(0, -y * y_hat). However, since the target values are not bounded (they could be anywhere between -inf to inf), predicting larger absolute values is penalised more strongly than small absolute values, yielding very poor results.

  • $\begingroup$ You have two objectives, and how you combine them matters. Are you trying to ensure that #2 holds for 100% of your predictions, and that #1 is the best (let us say least squares) solution that can be found given #2? Or are you simply trying to formulate an objective that is a weighted average of the two, where you can specify the weight? $\endgroup$ – jbowman Feb 4 '18 at 18:53
  • 1
    $\begingroup$ Strongly agree with @jbowman. For example, suppose target is 0.1, you have 2 predictions, prediction 1 is -0.1, and prediction 2 is 10. which is better? $\endgroup$ – Haitao Du Feb 4 '18 at 19:34
  • $\begingroup$ It does not have to hold on 100% of my predictions. It should be a weighted average, the weights being model hyperparameters. $\endgroup$ – Eduardo G Feb 9 '18 at 0:24
  • 1
    $\begingroup$ Is it possible that there is a more tangible, domain-specific metric that can be used to measure the performance of your model? For example, your description sounds similar to a typical "stock price prediction" problem, where you want to accurately predict but also avoid generating buy/sell signals the wrong way around. In that case, the best summary metric is actually backtested portfolio performance (i.e. the profit you can make based on the strategy that the model enables you to use). What you use a model for is often the best, most natural metric to use. $\endgroup$ – Chris Haug Apr 20 '18 at 15:31

have you thought about just adding cross entropy maybe? For example:

mean_square_loss = tf.losses.mean_squared_error(labels=labels,
cross_entropy = tf.losses.sparse_softmax_cross_entropy(labels=class_labels, 
loss = tf.add(mean_square_loss, cross_entropy)

Would have to add logits of shape [?,2] and class_labels representing the sign of the real label as element of [0,1).

  • $\begingroup$ The problem is that the targets are relative to the predictions of my model, so they are not known beforehand. $\endgroup$ – Eduardo G Feb 9 '18 at 0:26
  • $\begingroup$ Why does your answer use tensorflow? $\endgroup$ – Matthew Drury Mar 17 '18 at 0:35
  • $\begingroup$ I was experimenting with tensorflow at that time, didnt see Eduardo mentioning a specific library, so i just put together a minimal example using a library that i feel comfortable with. $\endgroup$ – Cebarik Mar 20 '18 at 11:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.