I have a large set of images that I want to classify as "happy" or "sad". Each image is also tagged with the distance from the camera to the object being photographed, but for complex reasons, I'm not allowed to use that in the ML models.

Just to see what would happen, I split my data into ten sets based on distance to object, setting it up so that each of the ten sets was of equal size. Then I generated models for each of the ten sets separately. So I got ten models m1,m2,...m10.

I then validated these ten models on the data that had been held out in each case. In fact, I validated each of the ten models on each of the ten holdout sets. I expected either (a) each of the ten holdout sets to be best predicted by the corresponding model (if the distance to the object "mattered") or (b) each of the models to perform comparably on each holdout set (if the distance to the object didn't matter).

What actually happened, however, was that each holdout set was best predicted by the model for an adjacent data set. Could be higher, could be lower, but as long as it was adjacent, it did much better than the data actually used to build the model. So holdout set 8 was best predicted by either m7 or m9, with m8 not doing nearly as well. And this was true for all of the holdout sets! The differences were profound -- the model as built might predict 50% of the holdout set correctly (50% of the images are each of happy and sad), while the "neighboring" models might predict 70% correctly.

I cannot for the life of me understand how this can happen. If the models were overfit, I would expect them to be terrible across the board. But they aren't -- far from it. And there is a very clear pattern, to boot.

Any ideas? Thanks!


closed as off-topic by mkt, Michael Chernick, kjetil b halvorsen, mdewey, Carl Feb 18 at 18:16

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This question appears to be off-topic because EITHER it is not about statistics, machine learning, data analysis, data mining, or data visualization, OR it focuses on programming, debugging, or performing routine operations within a statistical computing platform. If the latter, you could try the support links we maintain." – mkt, Michael Chernick, kjetil b halvorsen
If this question can be reworded to fit the rules in the help center, please edit the question.

  • 1
    $\begingroup$ So wait, you partitioned your data into 10 subsets, each containing an image with a different distance from the camera. Then you built 10 models, each on one subset? And then you used the other 9 subsets to make predictions for each model and test them? This makes no sense. Also, what algorithm did you use? $\endgroup$ – user2974951 Feb 11 at 13:00
  • $\begingroup$ @user2974951: (I think each model had 1/10 of all images which is probably more than 1 for training.) Why should this not make sense? I think this is a perfectly valid test for what is called ruggedness in analytical chemistry. $\endgroup$ – cbeleites Feb 12 at 12:40
  • $\begingroup$ Do I understand you correctly: set 8 contains all distance 8 images and holdout set 8 contains all except the distance 8 images? $\endgroup$ – cbeleites Feb 12 at 12:45
  • $\begingroup$ @cbeleites That's how I understood it. Basically created a different model for each different "factor". $\endgroup$ – user2974951 Feb 12 at 12:54
  • $\begingroup$ I would say that this is nonsense, how do you expect a model to predict something it has never seen before? As for why the adjacent set performed better then the set upon which the model was built on... I have no idea. It may be an anomaly. $\endgroup$ – user2974951 Feb 13 at 10:21

I eventually figured this out. It turns out that each of the different sets were carefully constructed to ensure that the training and validations sets were independent (duh), but that the mechanisms I used varied from one set to another. So while the training and validation sets for (say) set 1 were independent, the training set for set 1 was not independent of the validation set for set 3, leading to artificially high predictive value when the cross validation terms were computed. I'm sorry to have asked a question that you guys didn't have enough information to answer!


Not the answer you're looking for? Browse other questions tagged or ask your own question.