4
$\begingroup$

Does anyone know of any good example code illustrations for the rulefit Rule Based Learning Ensembles package? The documentation is incredibly lacking. I was guided to the package by this paper.

If anyone who is familiar with the model could give a brief interpretation of how the Rule based method works (and why it works well or not), it would also be appreciated. The results in the paper look very promising.

Also, are there any other alternative ensemble packages that perform as well or have similar functionality? I'm familiar with Caret, although, I don't know if any of the Caret methods are necessarily comparable to the above. For instance, Caret uses linear correlation for the variable importance selection, whereas the Rule Based Method seems to use interaction statistics and partial dependence functions and plots to determine importance.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
4
$\begingroup$

There are a few rule-based models available in R. They differ in how they work (and I assume you want to use them for classification and not regression).

I would start with Quinlan's book [1]. He goes into a lot of detail about how to make rules out of decision trees. Also look at Witten et al [2] to see how a lot of Quinlan's ideas have been extended (and these are implemented in RWeka: PART and JRip)

Then there is C5.0 which has tree- and rule-based versions, including boosting methods. Quinlan hasn't published much in a while but he did open-source his code recently. There are some subtle differences between what he describes in his book and what is currently in C5.0. The main changes are boosting and winnowing (AFAICT).

I just sent a book to Springer a few days ago that has a lot of detail on how this works, so hopefully that will be available soon if you don't want to sift thru a ton of C code.

There is an R package C50 om CRAN that we made from his GPL version of C5.0. There is an argument for rules (to get a rule-based model instead of a tree) and trials for boosting.

Max

[1] Quinlan, R. (1993). C4.5: Programs for Machine Learning. Morgan Kaufmann.

[2] Witten, I. H., Frank, E., & Hall, M. A. (2011). Data Mining: Practical Machine Learning Tools and Techniques, Third Edition (3rd ed.). Morgan Kaufmann.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for sharing (I will definitely look into the packages and book). To clarify, I'm specifically looking into the benefits gained by various tree ensemble methods. In the paper, they show very good out of sample results (very low error and variance of error compared to former ensemble tree methods). Are there any kind of comparative (Error) studies available with known data sets using the boosted C5.0? $\endgroup$
    – pat
    Commented Dec 20, 2012 at 20:12
  • $\begingroup$ No, very little has been published. Quinlan has said that there are some occasions that ensembling a tree might make it worse (Quinlan, J. (1996). Bagging, boosting, and C4.5). I didn't think that would happen but it does. Someone (Stan Young I think) had the opinion that ensembles don't help trees much when the predictors are uncorrelated (Although I've never tested that). $\endgroup$
    – topepo
    Commented Dec 20, 2012 at 20:27
2
$\begingroup$

Six years after the original post, but may be of use to others:

R package 'pre' fits prediction rule ensembles through the algorithm proposed by Friedman & Popescu (2008). It is available from CRAN; development version and example are also available from GitHub (https://github.com/marjoleinF/pre).

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.