3
$\begingroup$

When using LightGBM in classification problems it is possible to use monotonic constraints. In binary classification problems the interpretation is straightforward: "The probability of class (say) A must be a monotonic function of feature X".

But how does LightGBM implement monotonic constraints in a multiclass classification problem? Does it even make sense to talk about monotonicity when we have more than 2 classes? If so, what is the interpretation? LightGBM accepts monotone_constraints without any complaints and it also affects the predicted probabilities.

The following code illustrate the question:

Packages

import numpy as np
import pandas as pd
import lightgbm as lgb
import matplotlib.pyplot as plt

Data

from sklearn import datasets
iris = datasets.load_iris()
X = iris.data[:, :1].copy()  # we only take one feature: "Sepal length".
y = iris.target

Binary Model

# Binary Model
y_binary = y[y<2]
X_binary = X[y<2,:]

model_binary = lgb.LGBMClassifier().fit(X_binary, y_binary)
model_binary_monotone = lgb.LGBMClassifier(monotone_constraints=[1]).fit(X_binary, y_binary)

X_plot = np.linspace(X[:,0].min(), X[:,0].max(), 1000)[:,None]
fig, axs = plt.subplots(1, 2, figsize=(2*5, 5))
for k, name in enumerate(model_binary._classes): 
    axs[k].plot(X_plot, model_binary.predict_proba(X_plot)[:,k], label="Standard")
    axs[k].plot(X_plot, model_binary_monotone.predict_proba(X_plot)[:,k], label="Monotone")
    axs[k].set_title(name)
axs[k].legend()

enter image description here

Multi Class Model

model = lgb.LGBMClassifier().fit(X, y)
model_monotone = lgb.LGBMClassifier(monotone_constraints=[1]).fit(X, y)

fig, axs = plt.subplots(1, 3, figsize=(3*5, 5))
for k, name in enumerate(model._classes): 
    axs[k].plot(X_plot, model.predict_proba(X_plot)[:,k], label="Standard")
    axs[k].plot(X_plot, model_monotone.predict_proba(X_plot)[:,k], label="Monotone")
    axs[k].set_title(name)
axs[k].legend()

enter image description here

$\endgroup$
6
  • 1
    $\begingroup$ In some cases, monotonicity seems to make sense. For instance, we might classify pictures as elephants, cats and dogs - I would assume the elephant probability to be a monotonic function of the amount of gray in the picture. As always, adding constraints can improve matters (as in regularization), but one can go astray. $\endgroup$ Commented Dec 5, 2022 at 13:42
  • $\begingroup$ Can you please use monotone_constraints_method='intermediate' or 'advanced'? It appears from your graph that the basic method doesn't really work in the case of multi-class tasks across all classes. $\endgroup$
    – usεr11852
    Commented Dec 5, 2022 at 14:28
  • $\begingroup$ (Also, absolutely use monotonicity if you expect it to hold, I have found one of the best ways to help model generalisation) $\endgroup$
    – usεr11852
    Commented Dec 5, 2022 at 14:29
  • 1
    $\begingroup$ @usεr11852 using monotone_constraints_method='intermediate' or 'advanced' does not seem to change anything significantly. I agree with your point about monotonicity and model generalization. I would like to use monotonicity, but I would also like to understand how it works (or not) in this example. $\endgroup$
    – BLaursen
    Commented Dec 5, 2022 at 14:54
  • 1
    $\begingroup$ Thank you for checking, in fairness, the elephant example absolutely explains why we might not be able to have multiple constraints. While "proportion of grey" is relevant for elephant vs others, it is irrelevant in cat vs others or dog vs others, which is actually what the other PDP are showing. And that is because we cannot have the same constraint for all three classes going the same way. Actually, yeah, that's the answer, if the constraint is the same for all classes (e.g. more grey increases the probability of class $i$ for all $i$) then it is infeasible. $\endgroup$
    – usεr11852
    Commented Dec 5, 2022 at 15:02

1 Answer 1

1
$\begingroup$

Monotonic constraints are not implemented for multiclass classification currently.

One of the reasons is that it is convoluted or even potentially infeasible to define a monotonic constraint moving in the same direction for all classes (as in the example shared). If anything such a choice would create a race condition such that the overall effect in class $i$ where the condition is least prominent than in class $j$, the relative pairwise PDP is negative.

The basic method is actually quite simple: We have a parent node that has minimum and maximum constraint values for our variable(s) of interest; say $x_1$ has $x_1^{\text{min}} =-1$ and $x_1^{\text{max}} =1$ . We perform a node split based on standard splitting criteria (e.g. loss function reduction). We calculate the new min/max constraint values for the left and right child nodes. If they respect our constraints (e.g. the left child node constraints are $x_{1L}^{\text{min}} =-1$ and $x_{1L}^{\text{max}} =0.4$ and the right child node constraints are $x_{1R}^{\text{min}} =0.45$ and $x_{1L}^{\text{max}} =1$, we accept the split; otherwise (e.g. if $x_{1R}^{\text{min}}$ was $0.35$) the split is rejected.

Particularly, for LightGBM this issue has been discussed in the GitHub issue here. That said, XGBoost has a very nice tutorial page on Monotonic Constraints that is fully applicable to LightGBM too.

The behaviour you see though definitely warrants an issue as it is buggy. It appears that LightGBM does try to enforce some monotonic constraints during training of multi-class case, the same way it would be at a binary case. Unfortunately, while in the case of a binary classification task, satisfying the constraints for the positive class automatically satisfies the constraint for the negative class, in the case of multi-class classification task, if the constraint is satisfied for one of the classes, it does not necessarily covers the other classes too.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.