7
$\begingroup$

I am facing a multiclass classification problem where I have 4 classes and one of them dominates over the others. I use a KNN classification model and the majority of the instances are being classified as the majority class. I used the weights = 'distance' parameter and it did improve, but not all what I expected. I know that adjusting the classification thresholds of each class can improve the classification in the classes with fewer instances, but I don't know how to do it. My code is this:

    import pandas as pd
    from sklearn.model_selection import train_test_split
    from sklearn.neighbors import KNeighborsClassifier
    from sklearn.metrics import accuracy_score
    from sklearn import metrics
    import numpy as np
    from sklearn.model_selection import cross_val_score
    from scipy.spatial.distance import braycurtis
    from sklearn.metrics import confusion_matrix


    df_X = pd.read_csv('df_data.csv')
    df_Y = pd.read_csv('df_Class.csv')

    X_train, X_test, Y_train, Y_test = train_test_split(df_X, df_Y, random_state=42)

    knn = KNeighborsClassifier(n_neighbors = 5, metric = braycurtis, weights = 'distance')
    knn.fit(X_train, Y_train)
    Y_pred = knn.predict(X_test)

    acc_score = accuracy_score(Y_test, Y_pred)
    print("Acierto de KNN en la partición de test:", acc_score)
    print(metrics.classification_report(Y_test,Y_pred))

    m_confusion = confusion_matrix(Y_test, Y_pred)
    print(m_confusion)

and my results are these:

               precision    recall  f1-score   support

           1       0.64      0.39      0.48       244
           2       0.77      0.49      0.60       371
           3       0.56      0.95      0.71       626
           4       0.64      0.34      0.44       408

    accuracy                           0.61      1649
   macro avg       0.65      0.54      0.56      1649
weighted avg       0.64      0.61      0.58      1649

    [[ 94   4 126  20]
    [ 21 182 127  41]
    [ 10   6 592  18]
    [ 23  43 204 138]]

Thank you very much!

$\endgroup$

2 Answers 2

3
$\begingroup$

In a binary situation, either the predicted probablity is above the threshold and corresponds to a categorical prediction of $1$, or the predicted probability is below the threshold, which means that the predicted probability of the $0$ category is above the threshold and corresponds to a categorical prediction of $0$. (Use some randomization rule for if the prediction is right on the threshold, which could be its own question.)

When you have three or more categories, it can be that none of the categories exceed the threshold. For instance, if you want to pick the category corresponding to a predicted probability exceeding $0.5$, it can be that each category is predicted with a probability no greater than $0.4$, such as $(0.4, 0.3, 0.2, 0.1)$ for a four-category problem. If I am going to use a threshold-based classification system, I would consider this a "no prediction" prediction, and my confusion matrix would not be square but would have another prediction category of "no prediction" in addition to the true labels.

The idea behind the software would be to get the raw model outputs, determine which category exceeds the threshold, and pick that category as the prediction. Since this is not a software page, I will leave the exact Python implementation to the OP, though a now-deleted answer seems to have a good idea.

Y_pred_proba = knn.predict_proba(X_test)[:,1]
Y_pred = (Y_pred_proba > threshhold).astype(int)

(If several categories exceed the threshold, I am not sure what I would do.)

$\endgroup$
4
  • $\begingroup$ I asked this question precisely on a software page and they recommended me to ask the question here. The idea you give me is approximately what I need, but I don't know how to apply it to programming. $\endgroup$ Commented Apr 7, 2023 at 16:21
  • $\begingroup$ The idea I'm looking for (and I don't know if it's right) is that, for example, for an instance the predicted probabilities correspond to one for each class, then the classifier (in my case knn) assigns the label with the highest probability, but the class majority has a greater chance of having a higher probability. And what I want is for the classifier to take into account a label so that it is assigned even if it is not the one with the highest probability. $\endgroup$ Commented Apr 7, 2023 at 16:22
  • $\begingroup$ I disagree with the SO comment that a question about software implementation belongs on Cross Validated. You seem to have solved the statistical/mathematical side and know what you want to do. Now it’s time to write software that does it, which is squarely within the purview of Stack Overflow. $\endgroup$
    – Dave
    Commented Apr 7, 2023 at 16:23
  • $\begingroup$ I also think that my question is within the scope of Stack Overflow, although I have received more answers on this page. Do you think my idea is acceptable for the machine learning part? $\endgroup$ Commented Apr 7, 2023 at 16:41
1
$\begingroup$

You divide predicted probabilities by class frequencies to create a class frequency-adjusted prediction, and your categorical prediction will be the class with the highest score. This gives you the same result as the threshold method proposed by Dave, but can be used even if multiple classes exceed the threshold.

E.g., your class frequencies are 0.8, 0.1, 0.1, and you get predictions for a sample 0.7, 0.14, 0.16; your adjusted score will be 0.875, 1.4, 1.6, and so the third class is your categorical prediction.

That being said, there are many problems with thresholding and classification, and in general, it is not recommended (at least on this site), and you will find several threads about it here. Mainly, with thresholded predictions it is much harder to detect if a model is better than another model, the thresholding usually ignores the actual costs of false positive and false negative predictions, and the estimated probabilities might be much more helpful for the actual user of a model than discrete predictions.

$\endgroup$
4
  • $\begingroup$ This seems to be realted to by idea here. // What if there are ties in the adjusted scores? // This does not seem to deal with changing the threshold. For instance, what if the threshold in your example is an adjusted score above $1.2$, so $20\%$ higher than the baseline frequency (prior probability)? $\endgroup$
    – Dave
    Commented Apr 6, 2023 at 12:11
  • $\begingroup$ then you do the same thing as people would do if there are ties in a binary balanced classification without any score adjustment, but I don't know what that is :) I think if you just want to calculate something from a confusion matrix, you can just give 0.5 and 0.5 to both classes, but I can imagine that not every software implementation would handle that and maybe there are other problems hidden in that method $\endgroup$
    – rep_ho
    Commented Apr 6, 2023 at 12:26
  • $\begingroup$ I don’t mean a tie. I mean two categories with predictions that exceed your threshold. $\endgroup$
    – Dave
    Commented Apr 6, 2023 at 12:37
  • 1
    $\begingroup$ in my example 2 categories exceed the threshold, you take the largest adjusted score $\endgroup$
    – rep_ho
    Commented Apr 6, 2023 at 12:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.