I have a neural network that is being used to classify three classes
-1, 0, 1. It going to be applied to time series data to determine a class at each time point to predict an event.
I have 26000 training samples per class with 25 features and I have justified there is no overfitting with 10 time K-Fold cross validation.
When I use data that isn't randomised and is actually part of the training set, it sometimes classified the complete opposite, and no matter how I tune the hyperparameters it always gets certain ones wrong.
I will show you two graphs.
The 0 in the x axis indicates start of an event and the orange line is what it should be. There will be a bit of lag and messy bits because the data is from a human and isn't perfect and I have smoothed the data, it is classifying every 10ms. The first classifies how I expect a 98.8% neural network to do so, and the other is just completely wrong!
What is happening here? How can this be possible for such a high accuracy neural network?
I appreciate your all your help :)