I understand bias and variance in the context of linear regression and cross validation. Is a similar notion relevant for binary classification ? If so, how may I calculate it ? Let’s say I have 1000 data points and I performing a 10 fold validation and I get TP, FP, TN and FN each time from a classifier. Is there a way to assess the bias and variance of the classifier ? Also classification is dependent on the decision threshold that’s been set. Given this, is the notion of bias and variance relevant to a classifier ?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
With respect to the question: "Is the notion of bias and variance relevant to a classifier?" an answer is not directly or accurately.
My rationalization is based on a statistic (the Gini coefficient) which is related to one of the more utilized metric in machine learning application, namely AUC, which stands for the area under the ROC curve.
Note, "ROC" equates to the area under the ROC curve also known as "receiver operating characteristic curve". As a good reference on the topic, I refer you to Wikipedia on ROC. Also, "AUC" refers equivalently to "Area Under Curve" and the "c-statistic" or the "concordance statistic".
Mathematically, Gini coefficient ($G_1$) is related to AUC by the formula:
${G_1 = 2 AUC - 1}$
Now, the term 'variance' in the current context is better replaced by the concept of relative mean absolute difference, as the latter is linearly related to the Gini coefficient. More precisely, the Gini coefficient as half of the relative mean absolute difference (the relative mean absolute difference is the mean absolute difference divided by the average to normalize for scale).
Now, with respect to bias, one could construct a known "confusion table" and proceed to simulate based on a select probability distribution and tabulate the computed Gini coefficient and compare to the known value based on the confusion table indicated ROC and AUC, and see if there is any discernible/problematic bias. In general, the literature suggests that the Gini coefficient is not unbiased.
