Loss vs. Classification Accuracy in applied problems

Question

In practical problems, where we want to for instance predict if a subject has a certain disease or not, we usually take classification accuracy as a measure which has the straightforward interpretation of the ratio of subjects diagnosed correctly. My question: Would the loss function give us any additional, useful information with this regards? E.g. has hinge-loss a useful interpretation when we base our predictions on Support Vector Machines (SVM)?

Edit: What I also do not yet fully understand: Is accuracy based on loss? For instance back to binary SVM, is predicting a new sample to belong to either group based on which loss would be minimal? And is this the so called "score" when for instance using fitcsvm in Matlab?

Answer 1

Accuracy is essentially the mean of the Losses under a zero-one loss function, so to answer your question, yes accuracy is just a loss function.

More specifically: For the Zero-one loss function is defined as:

$L(y,y^*) =\begin{cases} 0,& \text{if } y = y*\\ 1, & \text{otherwise} \end{cases}$

So the mean across all y is obviously the accuracy.

Answer 2

I want to argue that the premise of your question is flawed.

In practical problems, where we want to for instance predict if a subject has a certain disease or not, we usually take classification accuracy as a measure [...]

Maybe some people do, but I think there is a fair perspective that anyone doing this is not doing their job. It's certainly not an advisable practice.

If I'm building a system to predict whether someone has a disease or not, my primary responsibility is to think through how to operationalize these predictions. Really, I'm not actually building a system to make predictions, I'm building a system to advise a doctor on how to intervene. A prediction may be part of this system, yes, but it is not the whole system.

In constructing a decision rule for the doctor, I need to weigh the consequences of my advise: what are the consequences if I advise the doctor to intervene as if the patient has cancer when they in fact do not, what are the consequences if I advise the doctor to behave as if the patient does not have cancer when they in fact do, and so forth. The evaluation of my decision rule must take into account these costs. Accuracy is irrelevant. Not only irrelevant, but harmful in this case. A good decision rule will have poor accuracy.

So what of a model, how to evaluate that? The model should not predict whether a patient has the disease or not (again, accuracy is irrelevant), but should inform us of the probability the patient has cancer. Our intention may be to use this probability to construct the decision rule above, but the model used in constructing these probabilities should be evaluated on the basis of its job: do the probabilities faithfully reflect the true probabilities in the population.

This is a separation of concerns: models predict probabilities, decision rules tell us how to act (and should be informed by probabilities). Good separation of concerns maximize our flexibility in taking action, and provide maximal information about the situation. Probabilities are the one true path.

Answer 3

They are two different metrics to evaluate your model's performance usually being used in different phases.

Loss is often used in the training process to find the "best" parameter values for your model (e.g. weights in neural network). It is what you try to optimize in the training by updating weights.

Accuracy is more from an applied perspective. Once you find the optimized parameters above, you use this metrics to evaluate how accurate your model's prediction is compared to the true data.

Let us use a toy classification example. You want to predict gender from one's weight and height. You have 3 data, they are as follows:(0 stands for male, 1 stands for female)

$y_1 = 0, x_{1w}= 50kg, x_{2h} = 160cm$;

$y_2 = 0, x_{2w} = 60kg, x_{2h} = 170cm$;

$y_3 = 1, x_{3w} = 55kg, x_{3h} = 175cm$;

You use a simple logistic regression model that is $y = \frac{1}{1+e^-(b_1*x_w+b2*x_h)}$

How do you find $b_1$ and $b_2$? you define a loss first and use optimization method to minimize the loss in an iterative way by updating $b_1$ and $b_2$.

In our example, a typical loss for this binary classification problem can be:

$$-\sum_{i=1}^{3}y_{i}log(\hat{y_i}) + (1-y_{i})log(1-\hat{y_i})$$

We don't know what $b_1$ and $b_2$ should be. Let us make a random guess say $b_1$ = 0.1 and $b_2$ = -0.03. Then what is our loss now?

$\hat{y}_1 = \frac{1}{(1+e^{-(0.1*50-0.03*160)})} = 0.549834 = 0.55$

$\hat{y}_2 = \frac{1}{(1+e^{-(0.1*60-0.03*170)}}= 0.7109495 = 0.71$

$\hat{y}_3 = \frac{1}{(1+e^{-(0.1*55-0.03*175)}}= 0.5621765 = 0.56$

so the loss is $(-log(1-0.55) + -log(1-0.71) - log(0.56))$ = 2.6162

Then you learning algorithm (e.g. gradient descent) will find a way to update $b_1$ and $b_2$ to decrease the loss.

What if b1=0.1 and b2=-0.03 is the final b1 and b2 (output from gradient descent), what is the accuracy now?

Let's assume if $\hat{y} >= 0.5$, we decide our prediction is female(1). otherwise it would be 0. Therefore, our algorithm predict $y_1 = 1, y_2 = 1$ and $y_3 = 1$. What is our accuracy? We make wrong prediction on $y_1$ and y2 and make correct one on $y_3$. So now our accuracy is $\frac{1}{3}$ = 33.33%

PS: In Amir's answer, back-propagation is said to be an optimization method in NN. I think it would be treated as a way to find gradient for weights in NN. Common optimization method in NN are GradientDescent and Adam.

Answer 4

A related discussion can be found here: What are the impacts of choosing different loss functions in classification to approximate 0-1 loss

As mentioned by @Tilefish Poele Classification Accuracy is one type of Loss, which is 0-1 loss. There are other types of loss exist for different purpose.

I will give one example on why we need more loss functions and why accuracy is useless in some cases. Think about imbalanced classification, such as fraud detection. 99.9% of the time, a transaction is not a fraud transaction, so, simply predicting all transactions are not fraud will have very high accuracy. But such system is useless. This is why sometimes weighted loss are needed.

For hinge loss and logistic loss, one way of thinking is they are convex and can approximate 0-1 loss, which is none-convex and hard to optimize. Logistic loss has some probabilistic interpretations (maximize the likelihood estimation). But I am not aware any interpretation on hinge loss.