Suppose you are a machine learning researcher, someone came up to you showing a model he developed. He says:
- My model is doing much better than all existing models on predicting independent dataset. But...
- Out of tens of features, only three features are kept for the final model.
- The cross-validated training and testing accuracies are lower than the accuracy on independent data.
- The real mechanism behind problem is actually not known and seems complex. The independent data is published and commonly accepted. The training and testing data are lab-generated but not biased towards the independent data.
Would you believe the results? What things you would criticize?
As it looks like a homework, let me put a bit more details:
The published dataset size is too small to build statistical learning model to match the problem complexity. Therefore based on a hypothesis, a larger dataset is generated for building a machine learning system which will be evaluated by the published data. However the generated data are quite noisy with many potential false positives and false negatives. After feature selection, a small fraction of considered features are relevant for the model building but turns out to predict the independent dataset well. I tried hard to make all facts correctly done, but still would like to hear more critics.