The accuracy in ANN model's training set and validation set keep incerasing synchronistically I have two different datasets: A and B. I separated dataset A into A1 as a training set and A2 as validations set.
However, I find the accuracy of the two datasets keeps increasing when I train the model. The problem is the accuracy in test set B keep decrease. I think the data in A1 and A2 have the same distribution. So it is overfitting with our more training.
How could we deal with such a problem?
 A: What could be happening is hard to guess and a lot will depend on the specifics of your situation. Here are some possible ideas to investigate:

*

*Manually checking some examples is always a good idea. For example, this might help you to identify when the labels could be accidentally swapped around on dataset A vs. B. One can also look at some examples that the model gets very wrong, gets very right etc., which could also help find the occasional mislabeled example (probably not the main issue here though).

*Have you sanity checked that the pre-processing / prediction works exactly the same way for dataset B as for A2? Possible issues in this area could be inputting images with values on a 0 to 255 scale instead of 0 to 1, or dividing by 255 twice, mixing up indices (i.e. not matching the correct inputs to the correct label) and so on. It's a good idea to compare some inputs just before they are fed into the neural network to check they have the correct tensor shape and distribution of values. If you save and re-load your model, checking that that process worked without issues is also important.

*One option is that your training & validation split is not "clean". E.g. if you have data from multiple patients, and you have some records from the same patient in training and in validation, you could observe your situation. The issue here is that you get overfitting and the validation set does not protect you against it. A better validation set-up would be the answer.

*There could be target leakage. I.e. something in dataset A gives the answer away, but the same "trick" that the model learns does not work on dataset B. One famous example that has happened in practice was when images with the disease present have the diseased region highlighted, while those images without disease do not have such highlights. In that case a model may simply learn to detect highlights rather than spotting patterns that correlate with disease. Target leakage could also be a result of how the data was processed (there's infinitely many ways to get into this situation).

*Dataset B could be out-of-distribution vs. dataset A. I.e. when the inputs for B are too disimilar to the ones the model was given during training, then all bets are off on what a neural network will do. It may be hard to be sure when you are in this situation - perhaps it's obvious if you work with images and you can just look at some examples and see that they are completely different. If it's hard to compare manually, then looking at e.g. a UMAP projection of the final (or first flattened) hidden layer(s) and coloring them in by dataset (A2 or B) might highlight this.

*If applicable/possible, what does a very simple baseline model do? E.g. linear regression, logistic regression, or whatever is a very simple thing that is easier to check than a neural network...

There's probably a whole lot of other issues that could be in play, but these are some common ones I've seen before.
