3
$\begingroup$

I have followed some tutorial for classification using xgboost, the tutorial usually finishes at predict using the model, and evaluate its accuracy.

My question is, how do I extract the 'equation' out from the model?

In my example, it is a binary classification (1 or 0) from a feature list of 150 element, so I expect an equation of y = ax1 + bx2 + ... + zzzx150, or something similar, so I can actually use the model.

$\endgroup$
4
  • $\begingroup$ I think this should be left open and someone more expert than I at boosted trees could provide an answer as to why formula extraction is not really practical for boosted trees and some other ML tools. $\endgroup$
    – Peter Flom
    Nov 4 '18 at 12:12
  • $\begingroup$ so which type of model is practical for formula extraction? my difficulty is after following a lot of tutorial online, i can only see how a model is built, the prediction, but how to actually use it, doesnt seem to be the focus $\endgroup$
    – Victor
    Nov 4 '18 at 19:12
  • $\begingroup$ Linear regression for a start - and then any generalized linear model for a second $\endgroup$ Nov 5 '18 at 1:41
  • $\begingroup$ @XavierBourretSicotte i actually found a function of feature importance, i think this is what i need, isnt it? if i want to understand how the features contribute $\endgroup$
    – Victor
    Nov 5 '18 at 8:05
4
$\begingroup$

If you are using the R interface, there is xgboostExplainer.

xgboostExplainer outputs the contributions of the features to the predictions on a per-instance basis. In a binary classification task, for example, the prediction for an input vector $\left( x_1, x_2, \dots, x_M \right)$ is usually made based on a predicted "success" probability, $\hat{p}$:

$$ \hat{p} = \mathrm{sigmoid} \left( f_{1}\left(x_1\right) + f_{2}\left(x_2\right) + \dots + f_{M}\left(x_M\right) \right) $$

For simple logistic regressions, $ f_{i}\left(x_i\right) $ is a linear function of $x_i$, i.e. $ f_{i}\left(x_i\right) = w_i x_i$ and so we will be able to extract and interpret the estimates for $w_1$. For boosting trees, however, the $f_i\left(x_i\right)$ is generally assumed to be non-linear, so there is no "coefficients" or "formula" like those found in generalised linear models to speak of; while the leaves of the trees do have scores/weights assigned to them, they are not very straightforward to interpret.

Given a trained xgboost model and an input vector, xgboostExplainer will tell you the values of the individual $f_{1}\left(x_1\right),\ f_{2}\left(x_2\right),\ \dots,\ f_{M}\left(x_M\right)$ for the said input. These can be interpreted as the "contributions" of each feature towards its final prediction. You may then interrogate the explanations produced for your predictions to understand the (non-linear) relationships learnt by the model. See this article for a nice exposition of this idea and the package's capabilities.

For the python API, there is the ELI5 package; I haven't used it myself but judging by its documentation it is similar to xgboostExplainer but covers many different types of models (including xgboost).

$\endgroup$
3
  • $\begingroup$ is it actually similar to feature importance that you can find in python (sklearn)? $\endgroup$
    – Victor
    Nov 5 '18 at 8:03
  • $\begingroup$ @Victor not exactly. Roughly speaking, the feature importance metrics from sklearn are tied to the model; they describe which features have been most informative to the training of the model. The explanations produced by the xgboost and ELI5 are for individual instances. For generalised linear models (e.g. logistic regression), one can derive the importance for individual instances by extracting the weights (say $w_1$ from the model and combine them with the values of input features (say $x_1$, to get $w_1 x_1$). This is not possible, however, for boosting trees, such as xgboost. $\endgroup$ Nov 5 '18 at 9:24
  • $\begingroup$ It is need to be said that this does not give the "formula" of the model, but rather tries to give an approximate answer on why the model gave its answer. The described functions have nothing to do with "using the model", but rather trying to understand it. $\endgroup$
    – Tim
    Nov 11 '18 at 12:17
3
$\begingroup$

My question is, how do I extract the 'equation' out from the model

It all depends on the model you are using. Some models do not have a straighrforward formula you can extract, and are often called blackbox models. I presume you are looking for some form of interpretability of your model... Since you are using an XGBoost model then the answer is not straightforward. You could try the following:

  • Extract the first decision tree and use it as a basis for extracting some form of equation (in the shape of a decision tree) - but this won't be as useful as the decision tree from a standard decision tree model due to the bagging and boosting effect of XGboost
  • Extract the weights of the different trees, and then compare it with the structure of the trees. But again you won't get any straightforward equation
  • Compute the partial dependency plots for a particular feature (or interaction of features)
$\endgroup$
2
  • $\begingroup$ I can understand your point, so how can I extract the weight and the decision tree please? Also what kind of model can give me such straightforward relationship? in and order word, the importance of each feature? $\endgroup$
    – Victor
    Nov 4 '18 at 1:13
  • $\begingroup$ look at the API of Xgboost there is a way to get the weights of each tree stump. There is also a method for getting the feature importance based on some pre-defined metric. In general I would advise following a full tutorial on the topic as your questions are too broad and generic for this site. $\endgroup$ Nov 4 '18 at 1:14
1
$\begingroup$

In my example, it is a binary classification (1 or 0) from a feature list of 150 element, so I expect an equation of y = ax1 + bx2 + ... + zzzx150, or something similar, so I can actually use the model.

Your question is ill-posed. The "formula" like $Y = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \dots + \beta_k X_k + \varepsilon$ defines linear regression model, where $Y$ is a linear function of the features $X_1,X_2,\dots,X_k$. There is no such simple formula for other models. For most of the other machine learning models, we also use features to predict $Y$, so we have something like $Y = f(X)$, but $f$ is a much more complicated, non-linear function.

If you'd like to code the predict(X) method by hand, that returns the outputs matching what XGBoost returns, you'd need to go deeper into the literature and source code of the implementation of the package that you used (since there may be minor differences between the papers and different implementations). Next, you'd need to basically re-write the appropriate XGBoost method yourself. Unless you want to do this "to learn more about inner workings of XGBoost", or you need to translate the code to different language, so that it works on your infrastructure, this isn't something that you should do. Every such implementation has some kind of predict(X) method that already does what you want to achieve, you should find examples in the documentation.

For more details on XGBoost, check the tutorial in their documentation or this talk by Tianqi Chen.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.