Which class to define as positive in unbalanced classification

Question

In a classification task, why do we usually choose the minority class as "positive" case for response variable? For example, if there are 1000, 9000 for class A and B respectively, we usually define class A as 1 and class B as 0, thus the response variable can have average as 0.1 instead of 0.9. If the response variable has average like 0.6 or 0.7, will there be any issues for the classification algorithm? Do we need to choose the class with average 0.4 or 0.3 as positive class? Thanks in advance!

Answer 1

It does not matter. It is done that way because it would be something like disease detection where most people don’t have the disease, but that’s what you want to catch. However, all of the math works out fine either way. If you made the heathy class the $1$s, your model would be outputting the probability of being healthy instead of the probability of having the disease. The English phrasing of those two might carry different connotations, but the mathematical logic is the same.

Answer 2

@Dave's answer is that all classification algorithms are symmetrical in regard to which class is called positive and which is called negative, or which is assigned to 0 and which to 1. This is probably true in almost all cases, but maybe one. It is true for decision trees and ensembles of decision trees since the classes are treated as categorical variables, as is for KNN, naive Bayes and so on. The two families of classifier algorithms that threat the classes as numbers, and therefore may be non-symmetrical in relation to the values 0 and 1 are logistic regression and SVM, but one can verify that the formulas are symmetrical, and thus no problem there.

But there is one case which I think may not be symmetrical regarding 2nd order aspects. I am not 100% sure and would appreciate if more knowledgeable people would chip in this discussion. Consider a neural network (MLP for example) optimizing log loss.

$$ log loss = \sum [y_i \log p_i + (1-y_i) \log (1-p_i)] $$

$y_i$ is the correct output, $p_i$ is the predicted probability.

The formula is symmetric by exchanging the y_i from 0 to 1 and vice versa, BUT the optimization being computed by the neural network is not (this is the 2nd order aspect). Output with $y_i = 0$ do not contribute to the log loss and thus do not contribute to the gradient and this it is not used in the gradient descent of the neural network. Therefore, by using the majority class 0 one is not using most of the data to optimize the MLP! The minimum is the same but a) if the gradient descent will reach that minimum, b) if it will the convergence rate to that minimum and c) the solution the MLP will settle given that it does not reach the global minimum are not the same whether one uses 0 or 1 for the majority class.

Am I right here?