# LightGBM interpretation of monotonic constraints in multiclass classification

When using LightGBM in classification problems it is possible to use monotonic constraints. In binary classification problems the interpretation is straightforward: "The probability of class (say) A must be a monotonic function of feature X".

But how does LightGBM implement monotonic constraints in a multiclass classification problem? Does it even make sense to talk about monotonicity when we have more than 2 classes? If so, what is the interpretation? LightGBM accepts monotone_constraints without any complaints and it also affects the predicted probabilities.

The following code illustrate the question:

Packages

import numpy as np
import pandas as pd
import lightgbm as lgb
import matplotlib.pyplot as plt


Data

from sklearn import datasets
X = iris.data[:, :1].copy()  # we only take one feature: "Sepal length".
y = iris.target


Binary Model

# Binary Model
y_binary = y[y<2]
X_binary = X[y<2,:]

model_binary = lgb.LGBMClassifier().fit(X_binary, y_binary)
model_binary_monotone = lgb.LGBMClassifier(monotone_constraints=[1]).fit(X_binary, y_binary)

X_plot = np.linspace(X[:,0].min(), X[:,0].max(), 1000)[:,None]
fig, axs = plt.subplots(1, 2, figsize=(2*5, 5))
for k, name in enumerate(model_binary._classes):
axs[k].plot(X_plot, model_binary.predict_proba(X_plot)[:,k], label="Standard")
axs[k].plot(X_plot, model_binary_monotone.predict_proba(X_plot)[:,k], label="Monotone")
axs[k].set_title(name)
axs[k].legend()


Multi Class Model

model = lgb.LGBMClassifier().fit(X, y)
model_monotone = lgb.LGBMClassifier(monotone_constraints=[1]).fit(X, y)

fig, axs = plt.subplots(1, 3, figsize=(3*5, 5))
for k, name in enumerate(model._classes):
axs[k].plot(X_plot, model.predict_proba(X_plot)[:,k], label="Standard")
axs[k].plot(X_plot, model_monotone.predict_proba(X_plot)[:,k], label="Monotone")
axs[k].set_title(name)
axs[k].legend()


• In some cases, monotonicity seems to make sense. For instance, we might classify pictures as elephants, cats and dogs - I would assume the elephant probability to be a monotonic function of the amount of gray in the picture. As always, adding constraints can improve matters (as in regularization), but one can go astray. Commented Dec 5, 2022 at 13:42
• Can you please use monotone_constraints_method='intermediate' or 'advanced'? It appears from your graph that the basic method doesn't really work in the case of multi-class tasks across all classes. Commented Dec 5, 2022 at 14:28
• (Also, absolutely use monotonicity if you expect it to hold, I have found one of the best ways to help model generalisation) Commented Dec 5, 2022 at 14:29
• @usεr11852 using monotone_constraints_method='intermediate' or 'advanced' does not seem to change anything significantly. I agree with your point about monotonicity and model generalization. I would like to use monotonicity, but I would also like to understand how it works (or not) in this example. Commented Dec 5, 2022 at 14:54
• Thank you for checking, in fairness, the elephant example absolutely explains why we might not be able to have multiple constraints. While "proportion of grey" is relevant for elephant vs others, it is irrelevant in cat vs others or dog vs others, which is actually what the other PDP are showing. And that is because we cannot have the same constraint for all three classes going the same way. Actually, yeah, that's the answer, if the constraint is the same for all classes (e.g. more grey increases the probability of class $i$ for all $i$) then it is infeasible. Commented Dec 5, 2022 at 15:02

One of the reasons is that it is convoluted or even potentially infeasible to define a monotonic constraint moving in the same direction for all classes (as in the example shared). If anything such a choice would create a race condition such that the overall effect in class $$i$$ where the condition is least prominent than in class $$j$$, the relative pairwise PDP is negative.
The basic method is actually quite simple: We have a parent node that has minimum and maximum constraint values for our variable(s) of interest; say $$x_1$$ has $$x_1^{\text{min}} =-1$$ and $$x_1^{\text{max}} =1$$ . We perform a node split based on standard splitting criteria (e.g. loss function reduction). We calculate the new min/max constraint values for the left and right child nodes. If they respect our constraints (e.g. the left child node constraints are $$x_{1L}^{\text{min}} =-1$$ and $$x_{1L}^{\text{max}} =0.4$$ and the right child node constraints are $$x_{1R}^{\text{min}} =0.45$$ and $$x_{1L}^{\text{max}} =1$$, we accept the split; otherwise (e.g. if $$x_{1R}^{\text{min}}$$ was $$0.35$$) the split is rejected.