Consider a multi-class problem with $c > 2$ classes. With this situation, the researcher is bound to deal with complication where there are prediction ties between classes. In my case, I've a support vector machine (SVM) that tries to classify observations in one of four classes [0, 1, 2, 3].

The SVM classifies by pinning one class against all the rest, therefore creating 4 decision functions. This strategy, as described by the documentation, is to build classifiers independent of other classes.

One-vs-the-rest (OvR) multiclass/multilabel strategy

Also known as one-vs-all, this strategy consists in fitting one classifier per class. For each classifier, the class is fitted against all the other classes. In addition to its computational efficiency (only n_classes classifiers are needed), one advantage of this approach is its interpretability. Since each class is represented by one and one classifier only, it is possible to gain knowledge about the class by inspecting its corresponding classifier.

See: One-vs-the-rest (OvR)

However, the problem that this type of classifier creates is that it creates the potential for conflicting predictions, in particular, situations where there are $p>2$ predicted classes. Conflicting predictions can arise in $c = 2$ problems, but I believe this is easier to resolve so I will not consider this situation in this post.

Below I present an example with code.

import numpy as np
import pandas as pd
from sklearn import svm
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_curve, auc
from sklearn.preprocessing import label_binarize
from scipy.spatial import distance
import matplotlib.pyplot as plt
import patsy
## read data and preprocess
df = pd.read_csv("https://raw.githubusercontent.com/vincentarelbundock/Rdatasets/master/csv/texmex/liver.csv", index_col=0)
df.columns = ["ALP_B","ALT_B","AST_B","TBL_B","ALP_M","ALT_M","AST_M","TBL_M","dose"] ## rename columns
df['dose'] = df.dose.map({'A':0, 'B':1, 'C':2, 'D':3}) ## map values to integers

## build design matrices
model_string = 'dose ~ ALP_B + ALT_B + AST_B + TBL_B + ALP_M + ALT_M + AST_M + TBL_M'
y, X = patsy.dmatrices(data = df, formula_like = model_string)

# shuffle and split training and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=.25, random_state=123)

model = svm.SVC(kernel = 'linear', C = 1, decision_function_shape='ovr') 
fitsvm = model.fit(y=y_train, X=X_train)

pred = fitsvm.predict(X=X_test) ## predicts class
test_acc = np.mean(y_test == pred)

y_score = fitsvm.decision_function(X_test)
pred_argmax = y_score.argmax(axis = 1) ## predict the class with the most highest dist value
print(np.mean(pred_argmax == pred )) ## 100% match

threshold = 0.1 ## some random thresholds
yhat = (y_score > threshold).astype(float)
yhat.sum(axis = 1) ## returns number of predicted classes per row/observation

In this example, you'll note that SVM uses the ovr strategy. The method fitsvm.predict(X=X_test) predicts ONE class for each test observation, but the way it resolves to do this is by simply using an argmax decision where it predicts the class with the highest decision boundary value. This is a fair approach, but probably not the most optimal.

One approach I use is to look at the false positive rate for each predictor (fpr), and choose the predicted class associated with the predictor with the lowest fpr. However, this is still not an optimal approach.

Thus, I am in search of methods for resolving ties between classes. Are there any "best practices" out there for breaking ties?

# Binarize the output
y_mat_test = label_binarize(y_test, classes=[0, 1, 2, 3])

def optim_thres(fpr, tpr, threshold):
    TOP = np.array([0,1])
    xy = np.array(list(zip(fpr, tpr)))
    dist = [(i, distance.euclidean(TOP, v)) for i,v in enumerate(xy)]
    dist.sort(key = lambda x: x[1], reverse=False)
    i, d = dist[0]
    return (fpr[i], tpr[i], threshold[i])

# Compute ROC curve and ROC area for each class
TOP = np.array([0,1])
fpr = dict()
tpr = dict()
thres = dict()
roc_auc = dict()
optim = dict()
for i in range(n_classes):
    fpr[i], tpr[i], thres[i] = roc_curve(y_mat_test[:, i], y_score[:, i])
    roc_auc[i] = auc(fpr[i], tpr[i])
    ## next find optimal thres
    optim[i] = optim_thres(fpr[i], tpr[i], thres[i])


lw = 2
axes = []
for i in range(n_classes):
    ax, = plt.plot(fpr[i], tpr[i], 
            lw=lw, label='ROC curve (area = %0.2f)' % roc_auc[i])
    _fpr, _tpr, _thres, = optim[i]
    label = "threshold: " + str(round(_thres, 4))
        xy=(_fpr, _tpr), xytext=(-20, 20),
        textcoords='offset points', ha='right', va='bottom',
        bbox=dict(boxstyle='round,pad=0.5', fc='yellow', alpha=0.5),
        arrowprops=dict(arrowstyle = '->', connectionstyle='arc3,rad=0'))
plt.plot([0, 1], [0, 1], color='navy', lw=lw, linestyle='--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver operating characteristic example')
plt.legend(loc="lower right")

ROC Plot with a threshold identified


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.