Neural network working well on datasets near the training set, but poorly on farther datasets. Why? I've been using a siamese neural network for the binary classification of biological data. 
Each entry of the datasets I'm using has a position coordinate.
My problem is that, even if my neural network is able to do excellent predictions on the datasets that are spatially near to the training set, it is not able to do the same on farther datasets.
I'm using an held-out (no k-fold cross validation) optimization approach: the algorithm reads an input dataset, and splits it into a training set containing the 80% of the input elements, and a validation set containing the remaining 20% of the input elements.
The algorithm trains the neural network by using the training set, and then applies the trained model on the held-out validation set. By doing this, the algorithm is able to get excellent prediction scores on the validation set (e.g. Matthews correlation coefficient >= 0.9).
On the contrary, the problem come up when I try to apply my trained siamese neural network to test sets that are NOT adjacent to the training set. In these cases, my prediction scores go very bad (MCC ~= +0.1).
I also attach this simple image to better explain my problem:

Can someone help me with this?
What should I do to solve this problem?
Thanks
 A: You must have some autocorrelation in your data. In most cases, if one ignores correlation structure in the data (pseudolikelihood), the effect is that the estimated error in the data is too small. Suppose you considered the weather on two consecutive days, they are far more likely to be similar than the weather on two randomly selected days in the year.
Basically, you have done the test/training selection incorrectly. You must select at random from an entire sample and not contiguous rows. This is why simple random sampling is unbiased but convenience sampling is. Sampling contiguous rows of data, which are ordered in some sense, is effectively convenience sampling.
The graphic that you have used should be a scrambling of different colors for each of the training/test/validation sets.
A: Hypothesis 1: You have applied cross-validation incorrectly.
The information encoded in position is somehow related to the outcome. To ameliorate this, you could try not selecting your sets to be adjacent but instead a random partition. That might be enough to "average out" the effect of position.
Hypothesis 2: By ignoring the data in position, you're discarding information.
But even better would be to leverage the information encoded in position as a feature of the model in some way. I don't know what form that would take here because I don't understand what problem you're trying to solve, but if you're prediction weather on the basis of day of the year, you would want to incorporate the knowledge that yesterday's and tomorrow's temperatures are good predictors of today's. In the weather analogy, your model is getting good marks for predicting tomorrow as similar to today, but failing at predicting three months ahead.
Hypothesis 3: You actually want nested cross-validation.
You've only one hold-out set, which is an interval of position. Instead, you want an additional $k$ randomly-selected partitions of position as hold-outs. This is the natural extension of hypothesis 1 (folds should be composed as random samples of observations, not convenient samples) to the notion of why cross-validation is desirable (use each observation to score the model, not just some observations).
A: I think you have an issue with the way the position is encoded, as above, but my take on it is a little bit different, in that I think you have it encoded as an absolute distance to a reference point. If so your NN will work well only within the boundaries of the farthest point. To obviate this problem, one solution will be make the position relative, say to the center of the test set or something like that. Let me know if this helps with the issue.
P.S. if you think this was irrelevant to your question I will be glad to delete it .. was just hoping it would be helpful in solving your problem
