I get confused when it comes to KNN, why exactly does increasing K increase bias and reduce variance

Correct me if I’m wrong

My knowledge, suppose we have a regression problem

If k=1

and our nearest neighbor has a value of 5, that means our bias is zero right , our predicted value will be 5, bias=5-5,

For variance if a different data set was used and the value was 7, our variance would be 2,(high variance)

If k=3 and have values of 4,5,6 our value would be the average And bias would be sum of each of our individual values minus the average

And variance , if a different data set Witt values 4,6,7, average would be 5.667 and variance would be 0.6666( low variance)


1 Answer 1


Welcome Chukwudi. The definitions can be a bit confusing, but it looks like you are thinking of bias and variance as properties of the numerical estimates themselves instead of as properties of the estimator (i.e. of the function or algorithm that produces the numerical value of the estimates). I suggest you look at the mathematical definitions of these terms again before continuing on, paying attention to the notation. It's super important in stats to understand the notation and what is random and what is not. Notation will usually indicate what the "truth" (not random... for now, at least) is, what is observed data (random and not random), and what is an estimator (most always random, because they are functions of random variables).

  1. It might be more intuitive to think of bias as "on average, how wrong will my numerical estimate be, compared to the true value if I use this method/estimator" and variance as "on average, how much will my numerical estimate change if I use this method/estimator on different datasets."
  2. Again, remember that both bias and variance are expected values (i.e. averages over all datasets), and that bias is the average of the true value minus the estimate. In your examples above, you have not assumed any true values that I can see.
  3. Even if you had assumed true values, your examples are not a fair comparison. It's not a fair comparison to look at the bias of one estimator (your 1-NN example) versus the average bias of three estimators (your 3-NN example). To do this kind of numerical comparison fairly, you need to 1) apply both algorithms to the same dataset (y and x values), compare each estimate to the estimate from the other algorithm, then generate new data with the same true values and variance, and do it again, and again...

A few quick examples to clarify and develop intuition:

For the following examples, assume that the true data-generating mechanism is $y_i = f(x_i) + \epsilon_i$, where $\epsilon$ is just noise (i.e. mean zero, but nonzero variance $\sigma^2$). Pretend that I am trying to estimate/predict a new $y_{new}$ associated with a new $x_{new}$. Bias then is defined as $E[\hat{y}_{new} - f(x_{new})]$.

Consider a (dumb) estimating function/estimator, $\hat{y_{new}}=6$. That is, I don't care what the surrounding y values are, I don't care what the x value is, I just think it's 6. This will always be biased except when the underlying truth is coincidentally that $f(x_{new})=6$. But this estimator has 0 variance, because no matter what my data is, I still just estimate $\hat{y_{new}}=6$.

On the other hand, consider the estimator $\hat{y_{new}}=\bar{y}$ (n-NN regression). Now at least I'm learning a tiny bit from my data, so I'm probably less biased than having just chosen a random number such as 6.... but my bias is still large: for any prediction I try to make, I'm probably very biased unless somehow the true mean value of what I am trying to predict ($f(x_{new})$ is close to the overall true mean value of the observations in my data (average of all $f(x_i)$ for all $x_i$ in my data). And the variance of my estimator is now non-zero ($\sigma^2/n$).

Notice in both of the above cases, I only get small bias in particular locations or by dumb luck. I'm very likely to be doing a bad job of fitting most of the data. (i.e. underfitting / oversmoothing from oversimplified models).

Now consider 1-NN regression. Now I probably have pretty small prediction bias in most places - if I have any datapoint close to what I am predicting, I should do fairly well in terms of bias (assuming the function is reasonably smooth). However, the variance of my prediction is $\sigma^2$, since my prediction is just the value of the closest observation (which has variance $\sigma^2$, i.e. the variance is $n$ times larger than the n-NN case, and 2 times larger than the 2-NN case, 3 times larger than 3-NN, and so on).

Hope this helps!

  • $\begingroup$ You kind of lost me , in my example I have true values , 5 in the first case when k=1 $\endgroup$ Commented Sep 7, 2020 at 13:58
  • $\begingroup$ You confused me towards the end $\endgroup$ Commented Sep 7, 2020 at 13:59
  • $\begingroup$ Ok, I didn't know that you were saying that the true value of the new/predicted point is 5. The fundamental issue is that you're evaluating an algorithm based on 1 data point and 2 sets of "data" that are just drawn from thin air. Instead, try simulating data in your favorite software program using the general form I provided for you, i.e. create a true data-generating mechanism, and simulate data from it by adding noise, then generate predictions using each algorithm. Do that a bunch of times (like >500), and evaluate the bias and variance of the predictions from each algorithm. $\endgroup$ Commented Sep 8, 2020 at 23:26
  • $\begingroup$ As to "towards the end", which part? The variance calculations? This is fairly straightfoward if you assume uniform variance and independent observations. Try Googling something like "variance of sample mean derivation" if this is the part that was confusing. $\endgroup$ Commented Sep 8, 2020 at 23:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.