2
$\begingroup$

Consider a scenario where observed data is represented in grey and modelled data in red, as below

enter image description here

Here, the x-axis is a position, and the y-axis is an expected time, so that the slope defines, in a way, a speed.

The coefficient of determination, $R^2$, is commonly employed to evaluate the model's goodness-of-fit. However, I am interested in distinguishing between two specific sources of misfit, illustrated as follows:

  • Cases where the model fails to align with the data points but retains a local alignment in terms of the slope should not necessarily be considered a "misfit", as this might be a propagation of neighbouring dynamics (the slope in the data defines a speed, which is bounded)

enter image description here

  • Other instances where the model neither aligns with the data points nor matches the local slope

enter image description here

Could you suggest a mathematical method to quantify and differentiate these two types of misfits? What would be the optimal approach to assess them? Is there a better measure than $R^2$?

$\endgroup$

2 Answers 2

1
$\begingroup$

I think the usual measures of performance like mean squared error and mean absolute error do what you want (and, at least depending on the exact calculation, $R^2$ is just a deterministic function of mean squared error).

In the former case, mean squared error will apply a mild penalty. Ditto for mean absolute error. This is consistent with your observation that these misses aren’t particularly severe. At the same time, the predictions do miss the true values, leaving room for improvement.

In the latter case, where you see the predictions as particularly terrible, mean squared error and mean absolute error will impose severe penalties on the misses, consistent with your (reasonable) observation that the predictions are terrible.

$\endgroup$
1
  • $\begingroup$ Hm. I think that the MSE would be "too large" in the first example, because it measures the (squared) vertical distance between the two lines, and does not care about the fact that the first derivatives are quite similar. Because of the large slope, the MSE will be large although our intuition would suggest otherwise. An alternative might be a sum of square-integrated differences in the lines themselves, as the MSE, plus another term for the square-integrated (numerically evaluated) first derivatives. $\endgroup$ Commented May 3 at 15:40
0
$\begingroup$

There are other measures that you can consider, such as normalized Euclidean distance or Pearson correlation coefficient. To measure alignment or matching, you can use a Dynamic Time Warping algorithm to get DTW measure.

Dynamic Time Warping (DTW) is an algorithm that nonlinearly aligns temporal sequences in the time domain to determine a measure of their similarity independent of nonlinear variations in the time domain. Using DTW, similarity can also be detected between temporal sequences that vary according to speed. See the wiki article for more information.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.