My friend just exited an interview where he had used the terms "underfitting" and "overfitting" as equivalents to the bias/variance tradeoff, but he says his interviewers were looking more for an understanding of bias/variance tradeoff using those exact terms. As a trained statistician, he had used "underfit"/"overfit, which feel appropriate to me, but perhaps I am incorrect and need help from the community to get this straightened out. Here are my reasons as a statistician for using "underfit" and "overfit".
Error in a model is bias + variance + irreducible error. This last part often gets unnoticed when speaking about the bias/variance tradeoff. This assumes the world is stochastic and not deterministic.
When we speak of "high bias" and "low variance", this can be equivalent to "underfitting". When we speak of "high variance" and "low bias" this is equivalent to "overfitting".
Would we really ever know if we had high bias (or low bias), when we include the irreducible error into the equation? My mind immediately thinks that using the term "high" or "low" assumes we have made a normative judgment to the bias, which we cannot unless there's a way to quantify the irreducible error.
Would "high bias" then be relative to the proportion of "known" irreducible error? For example, the bias to irreducible error + bias + variance is 50% of the total Error term. Or it's 10%, etc.
The assumption is that a perfect model would have bias = 0, variance = 0, and have the leftover irreducible error.
Assuming my logic is correct, what we should be saying instead is "higher bias" and "lower bias", as it's relative to this irreducible error.
What does everyone think? Am I way off on this?