A recent question got me wondering about the current state of the art in Combined Multiple Models (CMMs). I am familiar with work by Domingos in the 90s, such as this ICML paper:

Domingos, Pedro. "Knowledge Acquisition from Examples Via Multiple Models." In Proceedings of the Fourteenth International Conference on Machine Learning. 1997.

It describes a way to make a single C4.5 classifier from a bagged ensemble of C4.5 classifiers, and the approach requires making many randomly generated examples $\overrightarrow{x}$ from the ensemble of C4.5 rules.

Has there been follow-up work on alternative methods of generating random examples? And particularly, has there been any focused on Decision Trees and Random Forests?

I've searched with Google Scholar and found one promising result:

Van Assche, Anneleen, and Hendrik Blockeel. "Seeing the forest through the trees: Learning a comprehensible model from an ensemble." Machine Learning: ECML 2007. Springer Berlin Heidelberg, 2007. 418-429.

The acronym CMM has been hard to search for, however, so I am turning to the wisdom of CrossValidated for additional references or summarization. Thanks.


2 Answers 2


In their KDD 2006 paper ("Model Compression"), Bucila, Caruana, and Niculescu-Mizil recommend simply using a large pool of unlabeled data, which is readily acquired in some applications, at least more easily than labeled data. But if that is not available, they suggest sampling from a non-parametric estimate of the unlabeled data density. They propose a way to do this sampling without first forming the density estimate.


I'm a bit (hum!) late, but you may look at (chronologicaly) : Robert D Gibbons, Giles Hooker, Matthew D Finkelman, David J Weiss, Paul A Pilkonis, Ellen Frank, Tara Moore, and David J Kupfer. The computerized adaptive diagnostic test for major depressive disorder (cad-mdd): a screening tool for depression. The Journal of clinical psychiatry, 74(7):1{478, 2013.

Yichen Zhou and Giles Hooker. Interpreting models via single tree approximation. arXiv preprint arXiv:1610.09036, 2016.

Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. 2018. A Survey of Methods for Explaining Black Box Models. ACM Comput. Surv. 51, 5, Article 93 (August 2018), 42 pages. (especialy the table on page 93:26)

To be a little bit more specific about the oracle methods : among the authors referenced by Guidotti & alii above, Craven [1996, 2003] and Melville & Mooney [2006] uses gaussian distribution postulating zero covariance between features, but generating data as they build the tree (different estimation of variance for different branches of the tree). It's also possible to generate new data for the whole tree through a multivariate gaussian with mean and covariance matrix estimated on the whole training set, but the number of needed data quickly becomes very large. One may limit that problem by performing PCA on the training set and drawing the new data on a lower dimension subspace. One may also use deterministic methods. For instance (inspired by the Munge method from Bucilua & alii 2006) taking the middle point between every pair of data point in the training set (which obviously generate n^2 points if n is the size of the training set). For those interested (and for those wiling to correct my mistakes) I've given it a couple try and loaded the notebooks on Github here https://github.com/ljmdeb/GSTA


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.