Can we use glmtree for negative binomial distribution? I'm trying to create a conditional inference tree predicting number of root grafts in relation to distance between roots and number of roots. My variable response has a negative binomial distribution but in glmtree function there is no Negative binomial family. Is there an alternative in R ?
 A: No, glmtree() does not support negative binomial regression (at least not when the overdispersion parameter theta has to be estimated). But you can easily use the quasipoisson family which also adjusts for overdispersion.
Other alternatives would be to use a nonparametric ctree() instead of the parametric negative binomial approach. Possibly, a sqrt() transformation of the response helps.
Or you can roll your own model-based recursive partitioning algorithm using the general mob() function, see vignette("mob", package = "partykit").
As an example for the two parametric approaches, I'm using the RecreationDemand data from the AER package. And for the negative binomial regression I'm interfacing glm.nb() from MASS:
data("RecreationDemand", package = "AER")
library("partykit")
tr_qpois <- glmtree(trips ~ quality | ski + income + costS,
  data = RecreationDemand, family = quasipoisson, minsize = 30)

library("MASS")
negbin <- function(y, x, start = NULL, weights = NULL, offset = NULL, ...) {
  glm.nb(y ~ 0 + x, start = start, ...)
}
tr_negbin <- mob(trips ~ quality | ski + income + costS,
  data = RecreationDemand, fit = negbin, control = mob_control(minsize = 30))

I'm restricting the minimal sample size per leaf to 30 observations to stabilize the estimation of the negbin model. However, when fitting the tree and searching the split point some warnings are issued nevertheless. Also, the way glm.nb() estimates the theta parameter is relatively slow which is why the corresponding tree is rather slow.
The resulting tree structure is not identical but quite similar:
plot(tr_qpois)


plot(tr_negbin)


While mob() allows you to easily fit the tree, it does not have all the convenience features of plotting and predicting that glmtree() has, though.
