I have two datasets generated from two FPGA cricuits having almost same design. Both have
17 features as binary values where the last column is the class label
1. Each dataset has ~50K rows. The following is a snapshot of my data:
0,1,0,1,0,0,0,1,1,1,1,1,0,1,1,0,0 0,1,0,1,0,0,0,1,1,1,1,1,0,1,1,1,0 0,1,0,1,0,0,0,1,1,1,1,1,1,0,0,0,0 0,1,0,1,0,0,0,1,1,1,1,1,1,0,0,1,0 0,1,0,1,0,0,0,1,1,1,1,1,1,0,1,0,0 0,1,0,1,0,0,0,1,1,1,1,1,1,0,1,1,0 . . .
I use mlpclassifier in scikit-learn to perform classification using classical neural network. When loading a dataset, I split it into 80% training and 20% testing, and get very good accuracy of 98% in both training and testing accuracy -- No overfitting.
However, I noticed something unusual here. When I use one dataset as a training set and the other dataset as a testing set, I get a strange overfitting: training accuracy= 98% and testing accuracy= 48%, while if I split the same dataset into 80%/20% everything works well as mentioned above. Can someone explain this phenomena? How can I avoid this?
Thank you so much