0
$\begingroup$

The problem I'm working on is a multiclass-classification. Have been reading through lot of articles and documentation, but not able to figure out which of Accuracy_Score or Cross_Val_Score should be used to find the prediction accuracy of a model.

I tried using both but the scores are different. Cross_Val_Score() gave me 71% right prediction, but 69.93% using Accuracy_Score().

enter image description here

enter image description here

What could be the possible reason for mismatch?

Edit: Added Confusion Matrix and Classification report: enter image description here

$\endgroup$
1
$\begingroup$

They're going to be different because in cross_val_score, you obtain an accuracy for each of your folds and average them. For each CV fold, your training and tests set are different; so, you obtain different accuracy values for each of them, and it enables you to calculate standard deviation of your accuracies, which is enclosed in parentheses in your image. accuracy_score of sklearn.metrics library calculates the accuracy based on the inputs y_pred and y_true. For example, if you input your entire training set, you'll get accuracy of your entire training set, which is of course slightly different than your CV score.

$\endgroup$
  • $\begingroup$ Gunes, Thanks for the clarification. I see your point now. So, can we say accuracy_score is the correct way to measure overall prediction accuracy? $\endgroup$ – ranit.b Feb 23 at 14:23
  • $\begingroup$ It actually depends on what you're trying to measure, all metrics have some meaning, but for final evaluation of how good your model is, use accuracy_score with your test set. (Assuming you're not into precision/recall, or confusion matrix) $\endgroup$ – gunes Feb 23 at 14:26
  • $\begingroup$ Why do you say "not into precision/recall, or confusion matrix"? Yes, check them as well. Note, i've just added the confusion matrix and classification report. $\endgroup$ – ranit.b Feb 23 at 14:29
  • $\begingroup$ Because my comment could be misinterpreted for evaluating the model performance. It's highly valuable to check other metrics, especially in imbalanced datasets. $\endgroup$ – gunes Feb 23 at 14:32
  • 1
    $\begingroup$ No, in my example model performance is not good. Because, typically, in datasets with imbalance, people don't use accuracy for measuring performances. For your case, I wouldn't look at the accuracy directly, since your precision for class 3 is 0. There are more serious problems. And, I advise you to read through the answer here for a discussion on accuracy: stats.stackexchange.com/questions/312780/…, because this seems to be going towards another direction other than the main question posed here. $\endgroup$ – gunes Feb 24 at 10:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.