I'm working on a binary classification problem with class labels $Y \in {0, 1}$, and have a classifier that emits the probability $P(Y=1|X=x)$ for each test example $x$. To summarize the performance of the classifier on a validation set, I plot a histogram of predicted probabilities for each label, all on the same figure. An example probably illustrates this best.
import numpy as np
import seaborn as sns
from matplotlib import pyplot as plt
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X[:100], y[:100], test_size=0.5)
clf = LogisticRegression()
clf.fit(X_train, y_train)
class_probs = clf.predict_proba(X_test)
y0_preds = class_probs[np.where(y_test == 0), 1]
y1_preds = class_probs[np.where(y_test == 1), 1]
sns.distplot(y0_preds, color='blue')
sns.distplot(y1_preds, color='red')
plt.show()