A scenario of developing machine learning model Suppose you are a machine learning researcher, someone came up to you showing a model he developed. He says:


*

*My model is doing much better than all existing models on predicting independent dataset. But...

*Out of tens of features, only three features are kept for the final model.

*The cross-validated training and testing accuracies are lower than the accuracy on independent data.

*The real mechanism behind problem is actually not known and seems complex. The independent data is published and commonly accepted. The training and testing data are lab-generated but not biased towards the independent data. 


Would you believe the results? What things you would criticize?

As it looks like a homework, let me put a bit more details:
The published dataset size is too small to build statistical learning model to match the problem complexity. Therefore based on a hypothesis, a larger dataset is generated for building a machine learning system which will be evaluated by the published data. However the generated data are quite noisy with many potential false positives and false negatives. After feature selection, a small fraction of considered features are relevant for the model building but turns out to predict the independent dataset well. I tried hard to make all facts correctly done, but still would like to hear more critics. 
 A: I would be concerned about something called "data leakage".  Basically if you're trying to predict something but one of your features 'secretly' has more information than your model should know, that feature will become overly important.
For example, if one was doing a regression trying to predict home prices and in this simple example one of the features was property tax, that could end up being the most dominant feature.  For simplicity if it's one city say the property tax is 1%. A good enough model could determine that it only needs to multiply the property tax feature by 100 to get a nearly perfect (low error) result.
A lot of "too good to be true" scenarios in modeling come from data leakage.
That said, again using home price regression as the example, out of potentially several hundred features a few tend to be far more important than others. For example square footage, number of bedrooms, zip code, etc.
You also mention in the clarification that the published data isn't large enough to fit a model to. Depending on the number of samples in that data you may be able to do bagging on the published data to get a better idea of how your model/hypothesis are doing. (Since you mention in 3 that your accuracy is better on the real data than the synthetic data)
Ultimately the answer to "is this believable" would come down to understanding of the problem and data. 
