5
$\begingroup$

In practical problems, where we want to for instance predict if a subject has a certain disease or not, we usually take classification accuracy as a measure which has the straightforward interpretation of the ratio of subjects diagnosed correctly. My question: Would the loss function give us any additional, useful information with this regards? E.g. has hinge-loss a useful interpretation when we base our predictions on Support Vector Machines (SVM)?

Edit: What I also do not yet fully understand: Is accuracy based on loss? For instance back to binary SVM, is predicting a new sample to belong to either group based on which loss would be minimal? And is this the so called "score" when for instance using fitcsvm in Matlab?

$\endgroup$
  • 2
    $\begingroup$ In most problems you will be better off modeling probability (that is, risk) than with just classification see stats.stackexchange.com/questions/127042/… $\endgroup$ – kjetil b halvorsen Aug 28 '17 at 10:49
  • 1
    $\begingroup$ Thanks for the link. What I however do not yet fully understand: Is accuracy based on loss? For instance back to binary SVM, is predicting a new sample to belong to either group based on which loss would be minimal? And is this the so called "score" when for instance using fitcsvm in Matlab? Thanks $\endgroup$ – Pegah Sep 3 '17 at 13:25
  • 1
    $\begingroup$ Detecting rare classes is only a relevant problem if false negatives for the rare class are more expensive than false positives. Your classifier will not know this so you need to take precautions and not only optimize accuracy. $\endgroup$ – David Ernst Sep 4 '17 at 12:13
4
+50
$\begingroup$

Accuracy is essentially the mean of the Losses under a zero-one loss function, so to answer your question, yes accuracy is just a loss function.

More specifically: For the Zero-one loss function is defined as:

$L(y,y^*) =\begin{cases} 0,& \text{if } y = y*\\ 1, & \text{otherwise} \end{cases}$

So the mean across all y is obviously the accuracy.

$\endgroup$
  • $\begingroup$ True. Specifically in the case of hinge-loss: This loss function reflects w.r.t SVMs the condition of maximizing the margin, which does not (directly) reflect then calculated accuracies? $\endgroup$ – Pegah Sep 8 '17 at 8:36
  • 2
    $\begingroup$ I guess not since if you think about it you can hypothetically have say 9999 predictions being correct out of 10000 which would indicate under zero-one loss an accuracy of 99.99%, but if the error of the last incorrect score is absurdly large under hinge loss than you can have arbitrarily large loss. Generally they are strongly correlated though. $\endgroup$ – Tilefish Poele Sep 8 '17 at 8:47
4
$\begingroup$

I want to argue that the premise of your question is flawed.

In practical problems, where we want to for instance predict if a subject has a certain disease or not, we usually take classification accuracy as a measure [...]

Maybe some people do, but I think there is a fair perspective that anyone doing this is not doing their job. It's certainly not an advisable practice.

If I'm building a system to predict whether someone has a disease or not, my primary responsibility is to think through how to operationalize these predictions. Really, I'm not actually building a system to make predictions, I'm building a system to advise a doctor on how to intervene. A prediction may be part of this system, yes, but it is not the whole system.

In constructing a decision rule for the doctor, I need to weigh the consequences of my advise: what are the consequences if I advise the doctor to intervene as if the patient has cancer when they in fact do not, what are the consequences if I advise the doctor to behave as if the patient does not have cancer when they in fact do, and so forth. The evaluation of my decision rule must take into account these costs. Accuracy is irrelevant. Not only irrelevant, but harmful in this case. A good decision rule will have poor accuracy.

So what of a model, how to evaluate that? The model should not predict whether a patient has the disease or not (again, accuracy is irrelevant), but should inform us of the probability the patient has cancer. Our intention may be to use this probability to construct the decision rule above, but the model used in constructing these probabilities should be evaluated on the basis of its job: do the probabilities faithfully reflect the true probabilities in the population.

This is a separation of concerns: models predict probabilities, decision rules tell us how to act (and should be informed by probabilities). Good separation of concerns maximize our flexibility in taking action, and provide maximal information about the situation. Probabilities are the one true path.

$\endgroup$
  • 2
    $\begingroup$ I disagree. First, that's not what accuracy means (if I understand ou correctly), accuracy means the ratio of correctly predicted cases to the total number of cases, the OP even says so. Second, this is a point that I see so often misunderstood that it is worth taking a careful look at it. Statistical and ML model's job just simply is not to classify, it is to estimate probabilities. Taking that seriously maximizes our adaptability and knowledge of the situation. $\endgroup$ – Matthew Drury Sep 8 '17 at 4:05
  • 3
    $\begingroup$ @Pegah, you want to preserve the information in the probabilities up until the decision is made by the doctor facing the actual patient, when all (other) information is known & the costs of the different types of errors for that patient in that context can be guessed. It may help you to read: Classification vs. Prediction. $\endgroup$ – gung Sep 12 '17 at 2:27
  • 3
    $\begingroup$ @TilefishPoele, accuracy = (number of patients classified correctly) / (total number of patients); probability is equal to the long run proportion of patients with the disease at a given point in the covariate space. It may help you to read: Classification vs. Prediction. $\endgroup$ – gung Sep 12 '17 at 2:29
  • 2
    $\begingroup$ @Pegah, both are estimated from your sample, & both (you hope) apply to the population. If you fit, say, a logistic regression model, you can solve for the predicted value when X=xi. That prediction is on the scale of the 'linear predictor', but you can transform it into a predicted probability (this is very commonly done). The model is then predicting that, if you observed infinite patients, the proportion with the disease will match the predicted probability. There can be as many predicted probabilities as you may want from a given model. $\endgroup$ – gung Sep 12 '17 at 20:08
  • 2
    $\begingroup$ OTOH, if you dichotomize those predictions into 2 classes (diseased, healthy), you can compare those predicted classifications to the patients' actual statuses. Then you can compute the (single) accuracy from your model (when paired with your decision rule). It will be: (the number of patients classified correctly) / (the total number of patients). If you need more to understand the distinction b/t these two concepts, you should ask a new question. This really can't, & shouldn't, be explained in comments. $\endgroup$ – gung Sep 12 '17 at 20:12
2
$\begingroup$

A related discussion can be found here: What are the impacts of choosing different loss functions in classification to approximate 0-1 loss

As mentioned by @Tilefish Poele Classification Accuracy is one type of Loss, which is 0-1 loss. There are other types of loss exist for different purpose.

I will give one example on why we need more loss functions and why accuracy is useless in some cases. Think about imbalanced classification, such as fraud detection. 99.9% of the time, a transaction is not a fraud transaction, so, simply predicting all transactions are not fraud will have very high accuracy. But such system is useless. This is why sometimes weighted loss are needed.

For hinge loss and logistic loss, one way of thinking is they are convex and can approximate 0-1 loss, which is none-convex and hard to optimize. Logistic loss has some probabilistic interpretations (maximize the likelihood estimation). But I am not aware any interpretation on hinge loss.

$\endgroup$
2
$\begingroup$

They are two different metrics to evaluate your model's performance usually being used in different phases.

Loss is often used in the training process to find the "best" parameter values for your model (e.g. weights in neural network). It is what you try to optimize in the training by updating weights.

Accuracy is more from an applied perspective. Once you find the optimized parameters above, you use this metrics to evaluate how accurate your model's prediction is compared to the true data.

Let us use a toy classification example. You want to predict gender from one's weight and height. You have 3 data, they are as follows:(0 stands for male, 1 stands for female)

$y_1 = 0, x_{1w}= 50kg, x_{2h} = 160cm$;

$y_2 = 0, x_{2w} = 60kg, x_{2h} = 170cm$;

$y_3 = 1, x_{3w} = 55kg, x_{3h} = 175cm$;

You use a simple logistic regression model that is $y = \frac{1}{1+e^-(b_1*x_w+b2*x_h)}$

How do you find $b_1$ and $b_2$? you define a loss first and use optimization method to minimize the loss in an iterative way by updating $b_1$ and $b_2$.

In our example, a typical loss for this binary classification problem can be:

$$-\sum_{i=1}^{3}y_{i}log(\hat{y_i}) + (1-y_{i})log(1-\hat{y_i})$$

We don't know what $b_1$ and $b_2$ should be. Let us make a random guess say $b_1$ = 0.1 and $b_2$ = -0.03. Then what is our loss now?

$\hat{y}_1 = \frac{1}{(1+e^{-(0.1*50-0.03*160)})} = 0.549834 = 0.55$

$\hat{y}_2 = \frac{1}{(1+e^{-(0.1*60-0.03*170)}}= 0.7109495 = 0.71$

$\hat{y}_3 = \frac{1}{(1+e^{-(0.1*55-0.03*175)}}= 0.5621765 = 0.56$

so the loss is $(-log(1-0.55) + -log(1-0.71) - log(0.56))$ = 2.6162

Then you learning algorithm (e.g. gradient descent) will find a way to update $b_1$ and $b_2$ to decrease the loss.

What if b1=0.1 and b2=-0.03 is the final b1 and b2 (output from gradient descent), what is the accuracy now?

Let's assume if $\hat{y} >= 0.5$, we decide our prediction is female(1). otherwise it would be 0. Therefore, our algorithm predict $y_1 = 1, y_2 = 1$ and $y_3 = 1$. What is our accuracy? We make wrong prediction on $y_1$ and y2 and make correct one on $y_3$. So now our accuracy is $\frac{1}{3}$ = 33.33%

PS: In Amir's answer, back-propagation is said to be an optimization method in NN. I think it would be treated as a way to find gradient for weights in NN. Common optimization method in NN are GradientDescent and Adam.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.