6
$\begingroup$

I'm training a network to predict the angle of arrival of a signal. Labels are single values in the [-180, 180) interval.

I'm seeing a discontinuity in predictions around ±180 degrees, which makes sense as losses around that gap are incorrectly calculated by root mean square error.

I'm looking for a loss function that works in a modular way. A difference between 175 and -175 degrees should be calculated as 10 (instead of 350), if such thing exists.

It's my understanding that such a function introduces a discontinuity and thus may not be a valid approach. I'm looking for some guidance about how to deal with these kind of circular variables like angles, hour of the day, day of the week...

This has been addressed in the question "Encoding Angle Data for Neural Network", and I feel preserving linearity in angle variables is important (my input is also several angles), and I'm not getting good results with the sin/cos encoding approach proposed in that question. The problem is also discussed here: What is a correct loss for a model predicting angles from images?.

Here is what I'm doing currently, which works quite well with angles (-180, 180).

def metric_stddev_diff(y_true, y_pred):
    return tf.keras.backend.std(y_true - y_pred)

def model_create():

    model = tf.keras.Sequential([
        tf.keras.layers.Dense(128, activation='sigmoid', dtype='float64'),
        tf.keras.layers.Dense(64, activation='linear', dtype='float64'),
        tf.keras.layers.Dense(1, activation='linear', dtype='float64'),
    ])

    model.compile(optimizer='adam',  # 'rmsprop'  'adam',
                  loss='mean_absolute_error',  # 'mean_absolute_error'  'mean_squared_error'  'sparse_categorical_crossentropy'
                  metrics=['mean_absolute_error', metric_stddev_diff])

    return model
$\endgroup$
5
  • $\begingroup$ I kept using "mean absolute difference" which was working nicely in the angles case. I suspect that's wrong... I don't really know how to interpret this metric in the $\sin \theta, \cos \theta$ case and I guess the loss function is not valid. $\endgroup$
    – jjmontes
    Sep 6 '19 at 17:55
  • $\begingroup$ I think that working with points in the unit circle and using mse will yield better results. Instead of feeding the model with angles, use the points (cos(a), sin(a)). $\endgroup$
    – rbarreiro
    Jul 8 '20 at 9:35
  • $\begingroup$ Welcome to Cross Validated! What makes you think this? Do you have any references or simulations? How would you address the fact that this now is a multivariate regression with a response variable in the plane instead of the line? $\endgroup$
    – Dave
    Jul 8 '20 at 10:21
  • $\begingroup$ Could you please share your new custom loss function? One suggested above doesn't converge for me either. I'm training a CNN for learning 3 orientation angles. $\endgroup$
    – latida
    Feb 16 at 13:34
  • $\begingroup$ The one above is not any of the recommended solutions. Try the complex approach (sin/cos) for the input encoding, and for the loss function try using either MSE of the angles, or one of the loss functions suggested by @whuber below. My approach was simply an ad-hoc weighted sum of MAE of angle and magnitude which I cannot recommend as I cannot really explain formally why it worked. $\endgroup$
    – jjmontes
    Feb 16 at 15:49
11
$\begingroup$

Almost any loss function that is symmetric and differentiable at $0$ is locally quadratic. Thus, you don't have to be too fussy when searching for a good loss function when you need symmetry and differentiability.

Notice that with nearby angles $\phi$ and $\theta,$ the Taylor series expansion of the cosine gives

$$\mathcal{L}(\phi,\theta)=2(1 - \cos(\phi-\theta)) = (\phi-\theta)^2 + O((\phi-\theta)^4)$$

is locally quadratic at $\phi-\theta=0$ (and all integral multiples of $2\pi$) through third order. Moreover, this function of $\phi$ and $\theta$ isn't badly behaved: it's defined for all angles, is differentiable everywhere, and--most importantly--respects the modular nature of angle comparison. Thus $\mathcal{L}$ is a natural and simple angular version of a quadratic loss. This would be a good place to start your analysis.

If you need more flexibility, consider defining your loss as a function of $\sqrt{2(1-\cos(\phi-\theta))}:$ clearly this is a circular analog of the absolute difference.

$\endgroup$
3
  • 2
    $\begingroup$ +1 A possible additional point of interest regarding this loss function: Minimizing $\mathcal{L}$ (or equivalently $-\cos(\phi-\theta)$) is equivalent to maximizing the likelihood of a von Mises distribution, analogous to how minimizing the squared error is equivalent to maximizing the likelihood of a Gaussian distribution. $\endgroup$
    – user20160
    Sep 7 '19 at 18:22
  • $\begingroup$ I had realized that $(\phi-\theta)^2$ was the first term of the Taylor series but couldn't make such a connection to the cosine! However, I tried $1 - \cos(\phi-\theta)$ and it doesn't converge... maybe this approximation seems to only work well for small differences (as expected), whereas the output space spans the whole interval $[-\pi, \pi]$ :-? I am now using the complex representation with a custom loss that combines the absolute errors in angle and magnitude, and seems to behave much better near the limits and converge faster, while the absolute error is a bit larger for noisy datasets. $\endgroup$
    – jjmontes
    Sep 8 '19 at 14:42
  • $\begingroup$ The cosine, being the real part of a global analytic function $\exp,$ converges everywhere. $\endgroup$
    – whuber
    Jun 10 at 20:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.