6
$\begingroup$

I am using the glmtree function from the partykit package in R.

I would like to know how I can evaluate the models and how I can improve them.

I am growing a big tree (alpha = 0.9) and pruning with AIC as the criterion. I am using the AUC (pROC package) and the results are between 0.62 and 0.79.

fit <- glmtree(fD ~ 1 | Age + fGender + Qualification + fOccupation + SizeWorkplc,
  data = newdata, family = "binomial",
  minsize = 50, maxdepth = 4, alpha = 0.9, prune = "AIC")

prob <- predict(fit, newdata = newdata, type "response")
newdata$prob <- prob
g <- roc(fD ~ prob, data = newdata)
plot(g)

I am really new on this, so I would really appreciate some help.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
4
$\begingroup$

The strategy you describe looks very reasonable. For evaluation you can use the usual kinds of measures that you employ for other binary classiers (or trees in particular): misclassification rate (or conversely classification accuracy), log-likelihood, ROC, AUC, etc. Personally, I often use the ROCR package but the pROC package you used appears to offer useful tools for this as well.

For improving the model, you might consider whether extending the model part from an intercept (fD ~ 1) to something with a regressor. I would recommend to do so based on subject-matter knowledge which I presume you have for this analysis. If, for example, you suspect that the Qualification effect or the Age effect depends on interactions with the remaining variables than you could use fD ~ Age + Qualification | fGender + fOccupation + SizeWorkplc or something like this. The the choice of the model certainly depends on what you could interpret or which interactions you would want to assess.

Improve this answer
$\endgroup$
7
  • $\begingroup$ Thank you very much, Achim. Do you know how can I get confidence intervals for the glmtree predictions? $\endgroup$
    – Dani
    Commented Jul 10, 2015 at 1:46
  • $\begingroup$ With a little bit of glue code: For example with data("kyphosis", package = "rpart") you can fit m <- glmtree(Kyphosis ~ 1 | Start + Age + Number, data = kyphosis, family = binomial) and then predict(m, kyphosis, type = "response"). For the associated standard errors, you need a small helper function, say sefit <- function(object, newdata) predict(object, newdata, se.fit = TRUE)$se.fit that you can then pass on to predict(m, kyphosis, type = sefit). Whether or not these standard errors are really still valid (after selection of the tree), is a different question, though. $\endgroup$
    – Achim Zeileis
    Commented Jul 10, 2015 at 20:35
  • $\begingroup$ Thank you very much Achim for your help. Do you know if I can apply the glmtree to an ordinal variable? If yes, which family should I use? If not, do you know another tree-based model that I could use? Thank you once again! $\endgroup$
    – Dani
    Commented Oct 15, 2015 at 7:11
  • 1
    $\begingroup$ Ordinal data is not supported by glmtree(). We're currently working on a combination of the general mob() infrastructure with polr() and/or clm(). The fitting of the tree is realtively easy and straightforward but we haven't finished all the nice glue code for plotting and predictions etc. If you want to use a constant-fit tree (with only partitioning variables but no regressor variables), I would recommend to use ctree() which deals with ordered factor responses. $\endgroup$
    – Achim Zeileis
    Commented Oct 15, 2015 at 9:58
  • $\begingroup$ That would be great! Do you have any idea when it would be finished? Is it possible to fit a multinomial logistic with the glmtree()? I will try the ctree(). Thank you very much for your help! $\endgroup$
    – Dani
    Commented Oct 15, 2015 at 12:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.