As in non-directional statistics, seeking to summarize goodness of fit by scalar statistics or some significance test can be helpful, but more can be done. Competent statistical practitioners do not stop at $R^2$ for a regression-like model; they always want at the very least to look at the residuals too.
In general, suppose some model gives fitted or predicted directions to compare with observed directions. On the circle, the difference between two directions is reasonably the magnitude of the shorter rotation from one to the other, given a sign according to direction of rotation when appropriate. Concretely, and using degrees as units of measurement, the difference between $350^\circ$ and $10^\circ$ is just $20^\circ$, rotating clockwise, and that between $10^\circ$ and $350^\circ$ is similarly $-20^\circ$, where the minus sign marks anti- or counterclockwise rotation.
The distribution of these differences, themselves angles or directions between $-180$ and $180^\circ$ with a clear reference level at $0^\circ$, or the equivalent in any more congenial units (e.g. hours for time-of-day problems), is a simple reflection of the adequacy of any model. If various different models are being considered, comparing the pattern of these differences (which are just residuals) is usually straightforward.
Given directions collectively $\theta, \phi$ to compare (in circular statistics, trigonometric conventions frequently trump statistical conventions, so that Greek letters are used freely for variables):
Batschelet (1981, p.242) uses the notation |$\theta, \phi|$ for the absolute
value of this difference and points out that it is also arccos(cos($\theta - \phi$)).
The cosine of the difference varies between $1$ and $-1$ as the difference
varies from $0$ to $180^\circ$. It therefore measures similarity of angles and
its mean thus defines one kind of circular correlation. On the last
detail, cf. Batschelet (1981, p.182).
Another scale on which to measure difference is thus $1 -$ cos$(\theta - \phi)$.
Batschelet (1981, p.243) uses the notation $d(\theta,\phi)$. Yet another is
chord length $2$ sin$[(\theta - \phi) / 2]$. The latter two scales both yield
results which are $0$ when $\theta = \phi$ and $2$ when $\theta$ and $\phi$ are $180^\circ$ apart, but results coincide only at those two endpoints.
See also Mardia and Jupp (2000, e.g. p.18).
Batschelet, E. 1981. Circular statistics in biology. London: Academic
Mardia, K.V. and P.E. Jupp. 2000. Directional statistics. Chichester: