I have a large set of images that I want to classify as "happy" or "sad". Each image is also tagged with the distance from the camera to the object being photographed, but for complex reasons, I'm not allowed to use that in the ML models.
Just to see what would happen, I split my data into ten sets based on distance to object, setting it up so that each of the ten sets was of equal size. Then I generated models for each of the ten sets separately. So I got ten models m1,m2,...m10.
I then validated these ten models on the data that had been held out in each case. In fact, I validated each of the ten models on each of the ten holdout sets. I expected either (a) each of the ten holdout sets to be best predicted by the corresponding model (if the distance to the object "mattered") or (b) each of the models to perform comparably on each holdout set (if the distance to the object didn't matter).
What actually happened, however, was that each holdout set was best predicted by the model for an adjacent data set. Could be higher, could be lower, but as long as it was adjacent, it did much better than the data actually used to build the model. So holdout set 8 was best predicted by either m7 or m9, with m8 not doing nearly as well. And this was true for all of the holdout sets! The differences were profound -- the model as built might predict 50% of the holdout set correctly (50% of the images are each of happy and sad), while the "neighboring" models might predict 70% correctly.
I cannot for the life of me understand how this can happen. If the models were overfit, I would expect them to be terrible across the board. But they aren't -- far from it. And there is a very clear pattern, to boot.
Any ideas? Thanks!