0
$\begingroup$

I've used neural nets to learn a trajectory and the learned model is giving 'fairly reasonable' prediction when compared to my test data.

My question is on evaluating this model. I've seen Mean Squared Error (MSE) and Mean Absolute Error (MAE) as metrics for evaluating regression model performance. My concern is that both MSE and MAE calculate error by comparing predicted point to its respective test data point. In a case where one may see predicted data shift to the left or right of the test data, point by point comparison will show that there may be large error compared to test data, without considering the factor that they are similar if temporal alignment was done first. How could such a performance be captured and quantified? Image below shows that the model is predicting the trend well, but the onset of maximum value is shifted compared to test data. Black line is test data, red is predicted and grey is a small subset of training data. test data is in black, predicted is in red and training data set is in grey While doing a bit of search on this topic, I came across Edit Distance on Real signals (EDR) and Dynamic Time Warping (DTW) as techniques to measure similarity between sequences. My thoughts were to apply one of these techniques to determine similarity and use that in combination with MSE/MAE to quantify overall performance.

Any insight into well accepted techniques that can be used to evaluate performance that not only captures a "point-by-point" performance, but considers trend similarity as well would be greatly appreciated.

$\endgroup$
0
$\begingroup$

(minor note, DTW is not a metric, it is a measure)

Tony Bagnall and his team did 64 million experiments to answer such a question! https://arxiv.org/abs/1602.01711

Bottom line, DTW will either be the best, or within 1 or 2% of the best, for virtually any problem. See also this tutorial. http://www.cs.unm.edu/~mueen/DTW.pdf

Note that in your example, the gray and the red lines have different offsets, be sure to normalize your data.

$\endgroup$
  • $\begingroup$ Could you expand on your answer? This definitely sounds like a useful paper to link to, but it would also be helpful to have a slightly more detailed explanation here. $\endgroup$ – mkt Aug 1 '17 at 18:07
  • $\begingroup$ Great reference on DTW. Thanks for the link! Did you have any further thoughts on what some other well accepted metrics and/or techniques one may consider in this case to quantify performance? $\endgroup$ – MathQ Aug 1 '17 at 21:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.