I've used neural nets to learn a trajectory and the learned model is giving 'fairly reasonable' prediction when compared to my test data.
My question is on evaluating this model. I've seen Mean Squared Error (MSE) and Mean Absolute Error (MAE) as metrics for evaluating regression model performance. My concern is that both MSE and MAE calculate error by comparing predicted point to its respective test data point. In a case where one may see predicted data shift to the left or right of the test data, point by point comparison will show that there may be large error compared to test data, without considering the factor that they are similar if temporal alignment was done first. How could such a performance be captured and quantified? Image below shows that the model is predicting the trend well, but the onset of maximum value is shifted compared to test data. Black line is test data, red is predicted and grey is a small subset of training data.
While doing a bit of search on this topic, I came across Edit Distance on Real signals (EDR) and Dynamic Time Warping (DTW) as techniques to measure similarity between sequences. My thoughts were to apply one of these techniques to determine similarity and use that in combination with MSE/MAE to quantify overall performance.
Any insight into well accepted techniques that can be used to evaluate performance that not only captures a "point-by-point" performance, but considers trend similarity as well would be greatly appreciated.
