9
$\begingroup$

I am trying to understand the difference between bias-variance and overfitting-underfitting. If a modal overfits the data it means that it will not generalize well on new data because it over learns the training dataset. I can also say that this overfitting model is biased for the training examples. Now the question is, what is variance? Why does my overfitting modal has high variance when variance is not a model's property.

P.S. If I become able to make sense of the variance in terms of the model, I will be able to get bias in terms of the model as well.

$\endgroup$
1

4 Answers 4

10
$\begingroup$

First off: Bias and variance of a model are measures of how bad your model is, while over- and underfitting are possible reasons for why your model is bad.

Definition of bias and variance

Bias and variance are defined e.g. here. The idea is this: You want to learn some function $f(x)$. You have a lot of datasets $\{D_i\}_{i=1}^s$, $D_i = ((x_{i1}, y_{i1}), \ldots, (x_{in}, y_{in}))$, all of size $n$, and all drawn from the same population. Let's denote this set of datasets with $S$. Then you train your predictor $\hat f$ on each dataset $D_i$, so you end up with $s$ different predictors $\hat f_k, k=1,\ldots,s$.

Next, you fix an $x$ and you compute the sample mean and the sample variance of all your $s$ predictors on this single $x$: $$ \begin{align} m_{\hat f}(x) &:= avg_i(\hat f_i(x))\\ s^2_{\hat f}(x) &:= avg_i((\hat f_i(x) - m_{\hat f}(x))^2) \end{align} $$ Finally, you define the bias as the difference between $m_{\hat f}$ and the actual value $f(x)$: $$ bias_{\hat f}(x) = m_{\hat f}(x) - f(x). $$

Of course, the real value $f(x)$ is not known, so you cannot really compute the bias. Nevertheless, you can still reason about it.

(To be precise, the real bias and variance of the model at $x$ would only be obtained with $s\to\infty$.)

Note again, that this is dependent on $x$, i.e. each $x$ can have a different bias and variance. So we don't talk about the variance in a dataset, but about the variance of the predictions $\hat{f}_i$ at $x$ when learning from many different datasets $D_i$ from the same population. If you want to, you could average those over all $x$ which would then yield the average bias and variance of your model.

Bias-Variance Tradeoff

Often, but certainly not always, changes to your learning procedure that decrease the bias would increase the variance, and the other way around, hence the name bias-variance tradeoff. E.g., reducing the neighborhood size in nearest-neighbor methods would decrease the bias but increase the variance. However, increasing the size of your training dataset would often reduce both your model bias and variance.

Connection between over- and underfitting and bias and variance

Overfitting usually increases the variance, while underfitting increases the bias: Overfitting means that you have fitted an overly complex model to your training data, so a small variance in your training set can result in a large variance of $\hat f(x_{test})$ at your test data covariate $x_{test}$. Underfitting, on the other hand, occurs if you fit an overly simple model, that is not capable of adjusting to the peculiarities of your data. So, while $\hat f(x_{test})$ will not vary much at your test data covariate $x_{test}$ (because, being simple, it doesn't have the capacity to), it will often not be capable of "reaching" all points in your dataset, hence large bias.

$\endgroup$
2
  • 1
    $\begingroup$ @hasManyStupidQuestions I explain that the variance of a model is actually different from the variance of the data. I think that answers the question in the title. $\endgroup$
    – frank
    Commented Feb 15, 2022 at 4:06
  • $\begingroup$ I found it a little difficult to find that information in the answer, but you're right that it's there. $\endgroup$ Commented Feb 15, 2022 at 16:06
6
$\begingroup$

When we are building a machine learning model, we have to take two factors into account:

  1. On one hand, we want the predictions given by the model to be as accurate as possible, this means that we want the model to have as little error as possible.
  2. On the other hand, we know that the observations in our data set are likely to have errors: either measurement errors, or they are influenced by some aspect that we do not take into account in the model.

So with our model we want to achieve a compromise, we want it to be able to capture the general trend of the data, without being influenced by the specific errors of the set with which we train it. An example always helps,

enter image description here

  1. If our model is too general, we will be in the situation of the left image, underfitting, and not really capturing the behavior of the data.
  2. But if our model is not general enough, we will be in the situation of the middle image, overfitting, and we will be including in the model specific errors of the observations in the trainset. That is, a change in our data set will produce a very large change in the model.
  3. It is clear that the best model is the one in the image on the right, because it captures the general behavior (that curved trend) without being influenced by all the specific errors of the observations.

So finally, the variance in a model tells us how variable the model is. The larger the variance, the more abrupt changes, ups and downs we will see in the model's predictions. If the variance is very small, the model is very stable and under-fitted. But if it is too large, the model is too variable and over-fitted. We look for a middle ground between the two.

$\endgroup$
4
  • $\begingroup$ Just curious...I know this is more of a stats theory place, but what method did you use to get those fittings? I know how to do lm and loess in R, but the right two both look like loess at surface level. $\endgroup$ Commented Feb 14, 2022 at 8:32
  • 1
    $\begingroup$ I fitted these using splines $\endgroup$ Commented Feb 14, 2022 at 10:32
  • $\begingroup$ Thanks, ill check that out. $\endgroup$ Commented Feb 14, 2022 at 12:20
  • $\begingroup$ This doesn't seem to address the question in the title, even though admittedly it does address the question in the body. $\endgroup$ Commented Feb 15, 2022 at 2:44
1
$\begingroup$

The other answers makes a great illustration, thus I want to answer your "... when variance is not a model's property."

It can be a bit confusing since a model is not "data" as such, but a "thing" that produces data. But, since a model is used to produce data, when we talk about a "model having big variance" we actually, losely speaking, mean that "the data the model produces have a high variance".

Consider the following; say you have a model which always predicts the number 5 e.g

X = [1,2,3,4,5,6,7]
model(X) #[5,5,5,5,5,5,5]

No matter what data you use put into the model it will always output $5$, thus we would say that your model does not vary at all (eventhough it is actually the output that doesn't vary, but since the output is a function of the model, then the model doesn't vary at all either).

Now, say you have a model which does the exact opposite, it varies a lot

X = [1,2,3,4,5,6,7]
model(X) #[-12,0,30000,9,-10000,2500,13]

as you can see (relatively) small changes in your input data results in huge difference in your ouput data (the model has a big variance). With a good model, we would expect that inputs that are close to eachother would result in outputs that are close to eachother aswell, which is not the case here.

These two scenarios corresponds to underfit/overfit respectively (see @Álvaro Méndez Civieta great plots), where the overfitted-model jumps a lot (high variance in the output) and the underfitted model doesn't move at all (or very little).

$\endgroup$
2
  • $\begingroup$ The variance of a model is not calculated over different inputs, but over different training datasets. In fact, it's technically incorrect to talk about the variance "of a particular model". It's the variance of a "learning algorithm". $\endgroup$
    – isarandi
    Commented Feb 16, 2022 at 17:29
  • $\begingroup$ That is correct - but if one struggles with the variance/bias trade-off, it might be better to keep the explanation a bit simplified, but more understandable. Great comment tho! $\endgroup$
    – CutePoison
    Commented Feb 17, 2022 at 8:43
1
$\begingroup$

TL;DR the variance is computed over hypothetical other randomly sampled training datasets.

You can find the technical definition in the other answers, I try to give a bit of background understanding here.

Bias and variance (in this usage) ultimately come from a frequentist conceptual framework, which might be a bit counter-intuitive if you were taught in a Bayesian machine learning style (the opposite kind of problem also comes up, and perhaps even more frequently).

The main idea in frequentist statistics is to understand/imagine (training) data to be random, not fixed (and parameters fixed, but that's not important now). Many frequentist concepts in the end boil down to counterfactual thought experiments: what would be our result if our input data were different. If you come from a Bayesian perspective this may be strange, since the input data is what it is, and we want to train a model on this data, who cares what would happen if we had different data, right? In a way, yes, but it is still useful to think about whether our learning process is brittle and would output something very different if we had somewhat different training data.

For bias and variance, we imagine what would happen if you resampled a new training set, trained the model on that data and made your predictions anew. Rinse and repeat, and record what your outputs are and see how much spread (variance) they have as you vary the training data.

Outside of toy models that can be prodded and analyzed arbitrarily, all this is more of a thought experiment.

In the real world, your data is fixed, and you can't just sample arbitrarily many independent training sets to measure bias and variance. They are rather theoretical concepts to be aware of and terminology to use in communication about how your model is performing and what the reason for bad performance may be. Cross-validation is a common real-world approximation to analyze if your model is sensitive in this way to the exact training data. But the different CV folds are not independently sampled, so it's just an approximation.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.