0
$\begingroup$

My friend just exited an interview where he had used the terms "underfitting" and "overfitting" as equivalents to the bias/variance tradeoff, but he says his interviewers were looking more for an understanding of bias/variance tradeoff using those exact terms. As a trained statistician, he had used "underfit"/"overfit, which feel appropriate to me, but perhaps I am incorrect and need help from the community to get this straightened out. Here are my reasons as a statistician for using "underfit" and "overfit".

Error in a model is bias + variance + irreducible error. This last part often gets unnoticed when speaking about the bias/variance tradeoff. This assumes the world is stochastic and not deterministic.

When we speak of "high bias" and "low variance", this can be equivalent to "underfitting". When we speak of "high variance" and "low bias" this is equivalent to "overfitting".

Would we really ever know if we had high bias (or low bias), when we include the irreducible error into the equation? My mind immediately thinks that using the term "high" or "low" assumes we have made a normative judgment to the bias, which we cannot unless there's a way to quantify the irreducible error.

Would "high bias" then be relative to the proportion of "known" irreducible error? For example, the bias to irreducible error + bias + variance is 50% of the total Error term. Or it's 10%, etc.

The assumption is that a perfect model would have bias = 0, variance = 0, and have the leftover irreducible error.

Assuming my logic is correct, what we should be saying instead is "higher bias" and "lower bias", as it's relative to this irreducible error.

What does everyone think? Am I way off on this?

Thanks!!!

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

You have a good understanding of the bias variance tradeoff. I would think the overfitting and underfitting are caused by the bias and variance in the model as you explained it, i.e., bias and variance are not same as overfitting and underfitting, they are cause and effect.

The effect of underfitting or overfitting the data is caused by the bias and variance in the model.

All models are wrong, the error in the model is also something thats not under our control. The bias and variance are generally assumed to be independent to the irreducible error which is the reason for the decomposition of error to the three terms. Of course, the theory makes the assumption that we can decompose the error. In reality, we could spend a lot of time looking at the error and distributions of the data and parameters. :)

Improve this answer
$\endgroup$
5
  • $\begingroup$ If I hear you correctly, what we're saying is that when we use bias/variance in machine learning, it's with respect to the specific functional form of the model with this exact set of finite features that were considered. But, as we include more features, we tend to "overfit", etc. But again, it's with respect to THIS functional form. Not necessarily with respect to whatever the "actual/theoretical stochastic process that generates this data", which may be more in line with goodness of fit. $\endgroup$
    – DoctorDawg
    Dec 28, 2018 at 22:29
  • $\begingroup$ I agree, knowing the true/actual generating process is hard if not impossible for a stochastic system. I work with a specific hypothesis function, then fit the parameters of the function to best capture the data with the derived "features". Exploring the parameter space is harder than exploring the feature space. For example the "deep learning" stacked linear models framework explore the feature space really well. $\endgroup$
    – knk
    Dec 28, 2018 at 23:19
  • $\begingroup$ Here's a very simple case in point which may offer a better explanation to a possible misuse of the bias/variance tradeoff when we exclude irreducible error. Suppose one is training a linear regression model in R. In fact, you generated this yourself. Suppose you know the actual coefficients, now, because heck, you know how it was generated. It seems in machine learning that this may be a "high bias", "low variance" situation, when in fact, it's just the irreducible error that's left over. - And yes, the regression you trained on generalizes to any and all testing/out of time samples. $\endgroup$
    – DoctorDawg
    Dec 29, 2018 at 15:29
  • $\begingroup$ Why do you say its the "high bias" "low variance" situation, you are thinking of a specific generating function (possibly non linear) and using a linear model to generate fits? $\endgroup$
    – knk
    Dec 29, 2018 at 16:30
  • $\begingroup$ It seems like you are implying the bias variance trade off means ones of bias or variance has to be high and the other low, which isn’t necessarily true, you can have low variance and low bias and still observe the trade off. The trade off simply means that bias and variance tend in opposite directions, if you try to decrease bias it usually comes with an increase in variance. I don’t think whether you say high or higher really matters. At least this is my understanding $\endgroup$
    – astel
    Dec 29, 2018 at 16:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.