I have a data-set where customers reported their numeric preference for a sized product, comprised of ~600 samples with 15 different products. The data-set also includes 10 other numeric features, used to predict the size for each product. Sample size per product isn't huge, and ranges between 25-50.
Each product behaves a bit differently, so each gets its own ML model.
To perform model selection, I'm testing several regression models using a leave one out cross validation methodology. The best performing model type is selected for each product, then trained over the entire data-set.
The weird bit is this: When I use the entire data-set as a test set, the models which performed better during cross validation sometimes perform less well than another model which lost during model selection.
Since I'm using a leave one out for cross validation, I assumed results should differ that much between cross validation and testing on the entire data-set, but apparently that is not true.
I aware of the problem with testing with the entire data-set you used to train your models and the risks of over-fitting. I've done this in this case simply as a sanity check step rather than normal procedure.
I'd be happy to hear suggestions as to why this might be the case, at least for some of the products.