0
$\begingroup$

I have a dataset with ~800K samples, ~300 features and I'm trying to predict a binary outcome. I've started with sklearn's SGDClassifier (using log loss and l1 penalty), and I got a nice 0.67 auc score on my validation set; now I'm trying to improve this.

I have a reason to believe that different subspaces of features space "behave" differently, so I thought fitting a glmtree would make sense (splitting the features space to different subspaces and fitting a logistic regression in each of them). However, when trying to write the formula, I don't have any prior knowledge regarding which features should be used as predictors for the glm, and which should be used as conditioning variables.

Is there any automated way to select which is which?

$\endgroup$
1
$\begingroup$

glmtree() is particularly strong in situations where you have a GLM that you would typically fit to the whole sample, e.g., a voter targeting model (Rusch et al. 2013), a treatment effect model (Seibold et al. 2016), or an economic growth model (Wagner & Zeileis 2017). Then you can detect heterogeneity in the model parameters depending on the partitioning variables.

If you have no such "base" model and have no idea which variables are important, then you can fit an intercept-only GLM in each node. However, then the resulting tree is often rather similar to other classification trees (CART, CTree, etc.).

I have also seen situations where all numeric variables have been used in the regression and the categorical variables in the partitioning part. Possibly the numeric variables can also appear in both parts. However, this is only likely to yield good results if the number of numeric variables is small to moderate.

In your case with ~800K observations and ~300 variables and no prior information I personally would not bother with single trees anyway. I would probably use a random forest. You have enough observations to approximate potentially linear (or partially linear) effects sufficiently well through the forest.

References

  • Thomas Rusch, Ilro Lee, Kurt Hornik, Wolfgang Jank, Achim Zeileis (2013). Influencing Elections with Statistics: Targeting Voters with Logistic Regression Trees. The Annals of Applied Statistics, 7(3), 1612–1639. doi:10.1214/13-AOAS648
  • Heidi Seibold, Achim Zeileis, Torsten Hothorn (2016). Model-Based Recursive Partitioning for Subgroup Analyses. The International Journal of Biostatistics, 12(1), 45–63. doi:10.1515/ijb-2015-0032
  • Martin Wagner, Achim Zeileis (2017). Heterogeneity and Spatial Dependence of Regional Growth in the EU: A Recursive Partitioning Approach. German Economic Review. Forthcoming. doi:10.1111/geer.12146
$\endgroup$
  • $\begingroup$ Thanks for the detailed response! The reason I'm not using a RF approach is that I'm specifically looking for the partitioning rules; the ability to interpret the rules is roughly as important to me as the prediction accuracy. I understand from your answer that there's no principled way to select which is which - is that correct? $\endgroup$ – Adam Haber Oct 16 '17 at 13:11
  • $\begingroup$ I wouldn't know of a solution that will automatically and reliably separate linear, partially linear, nonlinear effects in the presence of high-order interactions and subgroup effects. There are search heuristics, however, e.g., performing AIC or BIC selection for the model in each node of the tree. But so far I have just tried these on a couple of data sets but never evaluated them properly. As for interpreting the rules: Note that these may change dramatically if you introduce linear effects in the subgroups! This model fits may still be similar but the structure could look very different. $\endgroup$ – Achim Zeileis Oct 16 '17 at 23:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.