What would be a good objective function for a computer vision model that predicts rotation? Let's say I have to correct images of scenes which can be rotated in the range $[0, 2\pi)$. For my particular application I already know a CNN makes sense. I'd use this to detect the orientation, then apply the appropriate correction. I'm just trying to think of how to frame the problem and objective function. This blog post mentions the use of categorical cross-entropy loss with a multi-label classification problem where the labels are $\{0, 1, 2, 3, ... 359\}$. To me this doesn't make much sense because if the target is 45, then 44 is penalised just as hard as 87.
Regression feels a lot more sensible, apart from the fact that here the number line wraps back around onto itself. But, since max-pooling works with back-propagation I don't see why I can't use a modified MSE like $min((y-\hat{y})^2, (y-\hat{y}-2\pi)^2, (\hat{y}-y+2\pi)^2)$.
Another potential solution could be to go for the categorical approach, but apply a gaussian smoothing to the target probabilities, centered around the target label.
Would anyone care to comment or advise?
 A: In circular statistics, your suggestion $\min((y−\hat{y})^2,(y−\hat{y}−2π)^2,(\hat{y}−y+2π)^2), $ which we could call arc distance loss, is actually one of the known loss functions that can be used. It works, and intuitively it is certainly more sensible than the categorical approach, for the reason you mention. The reason that it is still used, I suppose, is that it circumvents treating the predictions as circular entirely.
A possibly simpler and more common approach in the field of circular statistics is to use $1 - \cos(y - \hat{y}),$ which we can call the cosine distance loss, and which ranges between 0 (when $y = \hat{y}$) and 2 (when $y = \hat{y} + \pi).$
The arc distance loss is related to the Wrapped Normal distribution; the cosine distance loss is related to the von Mises distribution. Of course, other loss functions on the circle (and hypersphere, for that matter) are conceivable, and used, such as analogues of absolute error loss.
A: A common approach is to have the neural network output $u, v$ representing the angle $\hat \theta$ such that $\cos(\hat \theta) = \frac{u}{u^2+v^2}$, $\sin(\hat \theta) = \frac{v}{u^2+v^2}$. This avoids "boundaries" in the output which might prove challenging for backprop to learn.
Each rotation $\theta$ has a corresponding rotation matrix, which, conveniently, can be constructed with the sin and cosine of the angle.
$$R = \left[
\begin{array}{cc}
\cos \theta & -\sin \theta\\
\sin \theta & \cos \theta\\ 
\end{array}
\right]$$
Let $\hat R$ be the rotation matrix corresponding to the predicted angle, and $R$ be the true rotation matrix. Then $\text{tr}(R^T\hat R) = 2\cos (\hat \theta - \theta)$ is useful for computing the cosine distance loss. Conveniently, this trace trick also works for 3D rotations, even if the rotation happens along more than one coordinate axis.
