We have a legacy rule based model as a binary classier (0 vs 1). We want to replace it with a ML model. This rule based model uses only 4 features (age, race, gender, salary) and output 0/1. We have 20 extra features. So in total there are 24 features for 10 M unlabelled data points.

Here is our planned process:

  1. Use this rule based model to generate label for this 10 M data points (using only 4 features)
  2. Use ML models( like RF/logistic regression) to train on these data. The ML model will use 20+4 features to try to predict the label generated by rule based model


  • Will this ML model has any chance to outperform the rule based model?
  • Will it be able to "extrapolate" from label generated by rule based model? Or will it just use the extra 20 features to add noise?
  • Any document / site discussing about this approach?
  • $\begingroup$ So your features are age, race, gender.... I don't even want to know what you are trying to do... $\endgroup$ – Marsellus Wallace Jan 8 at 17:12
  • 2
    $\begingroup$ Machine learning can't squeeze blood from a stone. $\endgroup$ – gung - Reinstate Monica Jan 8 at 20:12

I would expect it to decide that only the 4 features of the original model matter and to come very close to completely matching the old model.

Clearly, a new model will never do better at predicting the output of the old model than the old model itself.

Unless you for it ground truths or at least information that is better than predictions of the old model, how could it outperform it on real outcomes (other than by pure chance)? If you had real outcomes available for this many cases, I would expect some state-of-the-art ML approach to do pretty well, if some the additional predictors matter at least a little bit and your training data is reasonably representative of the actual task.

  • $\begingroup$ The argument that ML model might outperform is because it uses all 24 features and may be able to extrapolate while rule base's model only use 4 features? $\endgroup$ – Harry Li May 7 '18 at 21:38
  • $\begingroup$ Let's assume that the original rule based model has high precision and low recall. Can the new ML model (trained on the previous system output but with a larger feature set) aim for higher recall at similar precision? $\endgroup$ – Marsellus Wallace Jan 8 at 17:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.