Model overfitting when using two separate datasets for train and test

Question

I have two datasets generated from two FPGA cricuits having almost same design. Both have 17 features as binary values where the last column is the class label 0 or 1. Each dataset has ~50K rows. The following is a snapshot of my data:

0,1,0,1,0,0,0,1,1,1,1,1,0,1,1,0,0
0,1,0,1,0,0,0,1,1,1,1,1,0,1,1,1,0
0,1,0,1,0,0,0,1,1,1,1,1,1,0,0,0,0
0,1,0,1,0,0,0,1,1,1,1,1,1,0,0,1,0
0,1,0,1,0,0,0,1,1,1,1,1,1,0,1,0,0
0,1,0,1,0,0,0,1,1,1,1,1,1,0,1,1,0
.
.
.

I use mlpclassifier in scikit-learn to perform classification using classical neural network. When loading a dataset, I split it into 80% training and 20% testing, and get very good accuracy of 98% in both training and testing accuracy -- No overfitting.

However, I noticed something unusual here. When I use one dataset as a training set and the other dataset as a testing set, I get a strange overfitting: training accuracy= 98% and testing accuracy= 48%, while if I split the same dataset into 80%/20% everything works well as mentioned above. Can someone explain this phenomena? How can I avoid this?

Thank you so much

Answer 1

What seems to be most likely is that the two datasets are not similar enough. As you say, they are generated from two different sources. The network is fitting specifically for one dataset, and not the other.

What you need is the network to generalize better. I see three different options:

1. You have a model for each network.

2. You merge the datasets and permute the data. This will force the network to generalize better.

3. You add a feature which has the same value for each dataset. Basically, this means a feature that contains the dataset id. This will also help generalization while letting the network do some "specialization" for each dataset.

How well 2 and 3 will work, is dependent on how similar/dissimilar your datasets are.