Definition of Bias and Variance in classification problems

Question

I was looking into a StatQuest video and he gave the meaning of bias and variance in regression problems Correct me if I’m wrong

1. Bias is the sum of squares error between the predicted and actual values In a data set, A low bias means the error is low, and it is accurately able to find the relationship between our x and y values ,
   A high bias means our error is high, and it is unable to accurately find the relationship between our x and y values, this is known as under fitting, it goes very badly on training data and test Data

2.Variance is the sensitivity of our model to different data sets

A low variance model, our line of fits wouldn’t be affected much by change in data sets, meaning the difference in y values for the different data sets wouldn’t be much , he used the case of having similar sums of square errors, meaning accuracy is similar and consistent

A high variance model, our lines of fits change a lot, across data sets, meaning the y values are far apart for different data sets, he used the case of having vastly different sum of square errors, meaning accuracy across different data sets was inconsistent, this is known as over fitting, it goes well on training sets but very badly on data sets

My issue comes with classification problems, how can I be able to use this StatQuest definition to explain bias and variance in KNN and Decision trees

I appreciate all answers, and would like an easy one as I relatively new to this field

Thank you for your answers in advance

Answer 1

When you look at a learning problem (classification being an particular example), you search for a function $\widehat f$ such that $\widehat{f}(X)=Y$ where $X$ is the features variable and $Y$ is the answer (label) variable (the class in classification). Using $\widehat f$, you try to approximate some $f^*$ which is most of the time the minimize of some loss function, for example in classification this is $$f^* = \arg\min_f \mathbb{P}(f(X)\neq Y).$$ So in fact, you use $\widehat f$ as an estimator of $f^*$. As such, the bias is a notion of distance between $\widehat f$ and $f^*$. This could be the squared distance or something but in fact given the problem, the bias will be better understood as $Bias = \mathbb{P}(\widehat f(X)\neq Y)-\mathbb{P}(f^*(X)\neq Y)$. This is a sort of approximation error. Here you have to understand that the larger the set $\mathcal{F}$ of function from which you choose $\widehat f$, the smaller it is. For example, in decision trees, if your tree is very deep then it is easier to approximate $f^*$.

Now, for the variance this is a bit more complex because in classification, the variance as you call it must also take into account the complexity of the set $\mathcal{F}$, for example you could define it with $\sup_{f\in \mathcal{F}}Var(f(X))$ but in fact there are better way to do this (see Rademacher complexity or VC dimension, but this is complex and outside the scope of this post). To give the idea, just think that the variance term will get larger and larger the more complex $\mathcal{F}$ is, it is linked to the fact that it is difficult to find a function in $\mathcal{F}$ if $\mathcal{F}$ is huge.

This is the reason why choosing for example the depth of a decision tree, or the number of nearest neighbors in KNN, is a bias/variance tradeoff.