I've been using a siamese neural network for the binary classification of biological data. Each entry of the datasets I'm using has a position coordinate.
My problem is that, even if my neural network is able to do excellent predictions on the datasets that are spatially near to the training set, it is not able to do the same on farther datasets.
I'm using an held-out (no k-fold cross validation) optimization approach: the algorithm reads an input dataset, and splits it into a training set containing the 80% of the input elements, and a validation set containing the remaining 20% of the input elements.
The algorithm trains the neural network by using the training set, and then applies the trained model on the held-out validation set. By doing this, the algorithm is able to get excellent prediction scores on the validation set (e.g. Matthews correlation coefficient >= 0.9).
On the contrary, the problem come up when I try to apply my trained siamese neural network to test sets that are NOT adjacent to the training set. In these cases, my prediction scores go very bad (MCC ~= +0.1).
Can someone help me with this? What should I do to solve this problem? Thanks