I want to explore using tree methods to condition a linear regression. Let's say the baseline regression is of the form $y = \beta x + \epsilon$.

In addition, I have conditioning data, a vector $c \in \mathbb{R}^k$, for every observation $(x, y)$.

I want to refine my regression model to be $y = \beta(c) x + \epsilon$. If I think of $\beta$ being piecewise-constant in tree-model like parition, then I want to partition the data based on the conditioner $c$ and then fit a different linear relationship in each partition.

For example, for scalar conditioning data, my tree might look like: For observations $(x, c)$, $y = \beta_1 x + \epsilon$ if $c > T$, $y=\beta_2 x+ \epsilon$ else.

Is there a well known statistical approach that I can use, especially one with an off-the-shelf method available in say the sklearn or xgboost packages?

I have toyed with the idea of fitting the $\beta$s using xgboost by redefining a custom objective function, $\hat{y}_i = \beta_i x$ for the xgboost leaf prediction $\hat{y}_i$ and then minimizing $(\hat{y}_i - y_i)^2$ as usual. Is this meaningful?


Off-The-Shelf Solution, in R

The cubist package by the legendary Max Kuhn does almost exactly this. Quoting from the documentation:

A tree is grown where the terminal leaves contain linear regression models. These models are based on the predictors used in previous splits. Also, there are intermediate linear models at each step of the tree.

It also includes a boosting-esque scheme called Ensembles By Committees, which can increase model accuracy. I have used cubist myself, and achieved accuracy similar to a Random Forest on my dataset. Your mileage may vary.

In Python

Based on some googling, you'll likely have to write it yourself, either by building decision trees and then extracting subsetting rules with which to fit linear models, or by porting/wrapping the C code that cubist is based on - available here. If I could volunteer a name, I'd suggest cubpyist. You might be able to join forces with this project, which seems to have just started.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.