0
$\begingroup$

In R I tried to measure the accuracy by performing a classification analysis using logistic regression analysis. I found that there are two ways to measure accuracy. One is AUC measurement using ROC curve and the other is accuracy using confusionMatrix.

When confusionMatrix is used, the probability exceeding 0.5 is regarded as 1, the value less than 0.5 is regarded as 0, and it is judged whether or not it matches the actual value.

If so, what should be considered as the model accuracy? Obviously, the values of AUC and confusionMatrix obtained from one algorithm are different. What is the different?

$\endgroup$
2
$\begingroup$

You can use many different metrics to measure "accuracy" for a classification model when comparing with other models - F score, G score, precision, recall, specificity, etc.

The AUC represents a metric which implies that for all values of a threshold, on average, model A will be better than model B if:

$AUC_{model A} > AUC_{model B}$

Selecting the threshold to classify a sample as 0 or 1 based on the probability predicted by the model, is an entirely different task from the prediction itself. In this case, you have selected 0.5 as the threshold. In most real-world classification tasks, the classes are highly imbalanced - i.e. 99% of the population belongs to class 0 and only 1% belongs to class 1. Hence, selecting 0.5 as threshold is not appropriate.

Several good answers on this forum have discussed why selecting thresholds should be avoided and best left as business decisions (see list of references at the end). However, in some cases this is unavoidable (e.g. text recognition where the character detected is required, not a probability score).

If the selection of thresholds is avoided, then you should use scoring rules for comparison. Refer to the wikipedia article on scoring rules.

If a threshold must be selected, then you should evaluate which metrics most closely optimizes the business problem you're trying to solve. Some classification related metrics are biased towards giving better scores for true positive rates and this may adversely impact the decision taken based on that score, so you need to choose the metric appropriately. Refer to the wikipedia article on metrics related to classification and Youden's J statistic.

You should also read these excellent discussions on CV related to these topics:

  1. RMSE (Root Mean Squared Error) for logistic models
  2. Obtaining predicted values (Y=1 or 0) from a logistic regression model fit
  3. Why isn't Logistic Regression called Logistic Classification?
  4. Logistic regression: maximizing true positives - false positives
$\endgroup$
  • $\begingroup$ Very good answer! $\endgroup$ – Matthew Drury Jun 9 '17 at 14:08
0
$\begingroup$

Given that you want to use logistic regression as a classifier - It would make sense to define accuracy in terms of correctly predicted (CP) observations. To evaluate model performance, you could then compare the % of CP with chance level (~50% in binary case). Because of the shape of the logistic curve, keep in mind that it becomes increasingly difficult to improve %CP,such that a 70-80% CP would already be a good perf.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.