3
$\begingroup$

Suppose you are a machine learning researcher, someone came up to you showing a model he developed. He says:

  1. My model is doing much better than all existing models on predicting independent dataset. But...
  2. Out of tens of features, only three features are kept for the final model.
  3. The cross-validated training and testing accuracies are lower than the accuracy on independent data.
  4. The real mechanism behind problem is actually not known and seems complex. The independent data is published and commonly accepted. The training and testing data are lab-generated but not biased towards the independent data.

Would you believe the results? What things you would criticize?


As it looks like a homework, let me put a bit more details:

The published dataset size is too small to build statistical learning model to match the problem complexity. Therefore based on a hypothesis, a larger dataset is generated for building a machine learning system which will be evaluated by the published data. However the generated data are quite noisy with many potential false positives and false negatives. After feature selection, a small fraction of considered features are relevant for the model building but turns out to predict the independent dataset well. I tried hard to make all facts correctly done, but still would like to hear more critics.

$\endgroup$
  • 1
    $\begingroup$ Is this homework? $\endgroup$ – Firebug Apr 23 '18 at 12:07
  • 1
    $\begingroup$ @Firebug, it's a problem I work on. Regarding the third point in my question, the feature statistics among training and independent data are actually consistent. If I explain that the higher accuracy is resulted from easiness of the data points, and the lower accuracy on training is partly due to noisiness in data, would you think this is a sensible model? I am looking for critics from professions. Thank you. $\endgroup$ – doubllle Apr 23 '18 at 12:33
3
$\begingroup$

I would be concerned about something called "data leakage". Basically if you're trying to predict something but one of your features 'secretly' has more information than your model should know, that feature will become overly important.

For example, if one was doing a regression trying to predict home prices and in this simple example one of the features was property tax, that could end up being the most dominant feature. For simplicity if it's one city say the property tax is 1%. A good enough model could determine that it only needs to multiply the property tax feature by 100 to get a nearly perfect (low error) result.

A lot of "too good to be true" scenarios in modeling come from data leakage.

That said, again using home price regression as the example, out of potentially several hundred features a few tend to be far more important than others. For example square footage, number of bedrooms, zip code, etc.

You also mention in the clarification that the published data isn't large enough to fit a model to. Depending on the number of samples in that data you may be able to do bagging on the published data to get a better idea of how your model/hypothesis are doing. (Since you mention in 3 that your accuracy is better on the real data than the synthetic data)

Ultimately the answer to "is this believable" would come down to understanding of the problem and data.

$\endgroup$
  • 1
    $\begingroup$ Great analysis! I do have an important feature which contributes ~50% by an importance ranking. Do you mean if one wants to prevent "data leakage", an overly important feature should be skipped? For binary classification, the decision boundary might be dominantly determined by this very feature. I think skipping important information is not very good. Maybe do some tricks like feature weighting? $\endgroup$ – doubllle Apr 24 '18 at 8:06
  • $\begingroup$ It really depends on your domain knowledge. It might make complete sense that one feature is super important relative to the others. Ideally you would understand why this particular feature is so important relative to the others. Going back to the house regression example, it's kind of intuitive that the size of the house or number of rooms would contribute greatly to the overall value. $\endgroup$ – PixelatedBrian Apr 24 '18 at 8:44
  • $\begingroup$ Also, you might want to experiment with modeling with the very good feature held out but adding in other features. If the model goes back to performing like other models historically then that might be an indicator that there's a problem. Or maybe you have just had a bit of a breakthrough. ;-) $\endgroup$ – PixelatedBrian Apr 24 '18 at 8:45
  • $\begingroup$ I did hold out the most important one, and tried intensively feature combinations. Without this feature, models performed significantly worse. Thanks a lot for the discussion. I accepted your answer. $\endgroup$ – doubllle Apr 24 '18 at 9:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.