I've been using a siamese neural network for the binary classification of biological data. Each entry of the datasets I'm using has a position coordinate.

My problem is that, even if my neural network is able to do excellent predictions on the datasets that are spatially near to the training set, it is not able to do the same on farther datasets.

I'm using an held-out (no k-fold cross validation) optimization approach: the algorithm reads an input dataset, and splits it into a training set containing the 80% of the input elements, and a validation set containing the remaining 20% of the input elements.

The algorithm trains the neural network by using the training set, and then applies the trained model on the held-out validation set. By doing this, the algorithm is able to get excellent prediction scores on the validation set (e.g. Matthews correlation coefficient >= 0.9).

On the contrary, the problem come up when I try to apply my trained siamese neural network to test sets that are NOT adjacent to the training set. In these cases, my prediction scores go very bad (MCC ~= +0.1).

I also attach this simple image to better explain my problem: Difference between validation set test and test set test

Can someone help me with this? What should I do to solve this problem? Thanks

  • 4
    $\begingroup$ The information encoded in position is somehow related to the outcome. You could try not selecting your sets to be adjacent but instead a random partition. That might be enough to "average out" the effect of position. But really you want to encode "position" as a feature, since it appears to have some bearing on the outcome. $\endgroup$ – Sycorax Apr 18 '16 at 19:43
  • 2
    $\begingroup$ I'm not sure what "position" means, but my observation is just that position is not ignorable -- if you ignore position, you lose important information about the outcome. Maybe it's as simple as giving each observation the position as an additional feature, taking values 1 to N. Maybe that's not valid for some reason pertaining to how your data were collected. $\endgroup$ – Sycorax Apr 19 '16 at 13:44
  • 1
    $\begingroup$ I see that you've added a bounty to this question. That's perfectly fine, but to attract a high-quality answer, you'll want to explain which suggestions in the comments did/did not work, so that people don't re-tread the same ground which you have, apparently, found unsatisfactory. $\endgroup$ – Sycorax Apr 20 '16 at 20:37
  • 1
    $\begingroup$ What kind of data is it, what do you classify, and what do you mean by position? In order to give a meaningful answer, it is necessary to understand how the position is related to the task. $\endgroup$ – user31264 Apr 20 '16 at 20:53
  • 1
    $\begingroup$ Then maybe this similarity depends on their position in the chromosome. You trained a neural network to detect region similarity in one part of the chromosome, but if the regions are in the different part of the chromosome, the rules are different. Just like if you train a network to recognize human faces, it might be unable to recognize cat faces. $\endgroup$ – user31264 Apr 21 '16 at 22:09

You must have some autocorrelation in your data. In most cases, if one ignores correlation structure in the data (pseudolikelihood), the effect is that the estimated error in the data is too small. Suppose you considered the weather on two consecutive days, they are far more likely to be similar than the weather on two randomly selected days in the year.

Basically, you have done the test/training selection incorrectly. You must select at random from an entire sample and not contiguous rows. This is why simple random sampling is unbiased but convenience sampling is. Sampling contiguous rows of data, which are ordered in some sense, is effectively convenience sampling.

The graphic that you have used should be a scrambling of different colors for each of the training/test/validation sets.

  • $\begingroup$ Thanks @AdamO, very clear. What do you mean with "scrambling of different colors for each of the training/test/validation sets"? Can you reproduce the image you're envisioning? $\endgroup$ – DavideChicco.it Apr 21 '16 at 23:08
  • $\begingroup$ @DavideChicco.it It means that in developing training / test / validation datasets, each row of observations should be equally and independently likely to be sampled into the respective dataset. In SAS, I would use the surveyselect procedure to randomly sample the 3 datasets. $\endgroup$ – AdamO Apr 21 '16 at 23:10

Hypothesis 1: You have applied cross-validation incorrectly.

The information encoded in position is somehow related to the outcome. To ameliorate this, you could try not selecting your sets to be adjacent but instead a random partition. That might be enough to "average out" the effect of position.

Hypothesis 2: By ignoring the data in position, you're discarding information.

But even better would be to leverage the information encoded in position as a feature of the model in some way. I don't know what form that would take here because I don't understand what problem you're trying to solve, but if you're prediction weather on the basis of day of the year, you would want to incorporate the knowledge that yesterday's and tomorrow's temperatures are good predictors of today's. In the weather analogy, your model is getting good marks for predicting tomorrow as similar to today, but failing at predicting three months ahead.

Hypothesis 3: You actually want nested cross-validation.

You've only one hold-out set, which is an interval of position. Instead, you want an additional $k$ randomly-selected partitions of position as hold-outs. This is the natural extension of hypothesis 1 (folds should be composed as random samples of observations, not convenient samples) to the notion of why cross-validation is desirable (use each observation to score the model, not just some observations).

  • $\begingroup$ Thanks @C11H17N2O2SNa. So, for your cross-validation in the Hypothesis 3, you're saying I should use say 10 held-out dataset segments, instead of using the same 10 dataset segments for training and for testing, as I would usually do for the classic k-fold cross-validation. Is this what you're saying? $\endgroup$ – DavideChicco.it Apr 21 '16 at 23:18
  • $\begingroup$ @DavideChicco.it That's correct. $\endgroup$ – Sycorax Apr 26 '16 at 14:47
  • $\begingroup$ Thanks @C11H17N2O2SNa. I don't know the nested cross validation very well; I tried to look for it on the internet but I did not find any clear explanation. Could you please signal to me a webpage or a resource where I can learn it? Thanks! $\endgroup$ – DavideChicco.it May 1 '16 at 14:25

I think you have an issue with the way the position is encoded, as above, but my take on it is a little bit different, in that I think you have it encoded as an absolute distance to a reference point. If so your NN will work well only within the boundaries of the farthest point. To obviate this problem, one solution will be make the position relative, say to the center of the test set or something like that. Let me know if this helps with the issue.

P.S. if you think this was irrelevant to your question I will be glad to delete it .. was just hoping it would be helpful in solving your problem

  • $\begingroup$ @davidchicco... when you get a chance .. out of curiosity can you post your final solution to the problem here $\endgroup$ – ashokragavendran Jun 2 '16 at 17:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.