If we have a classification/regression problem, when would we generally prefer to use families of models with high bias and low variance like multiple regression (logistic regression for classification)?
In other words, why would one use models with high bias and not use models with high variance and try to reduce variance?
Presumably your aim is to minimise out-of-sample prediction error or estimation error in some sense.
Here is a simple non-regression example:
Suppose you have a normally distributed random variable with unknown mean $\mu$ and variance $\sigma^2$, and you want to estimate $\sigma^2$ from a sample size $n$.
You decide to use some fraction of $\sum (x_i-\bar x)^2$, which has expectation $(n-1)\sigma^2$ and variance $2(n-1)\sigma^4$.
If you use as your estimator $s_k^2 = \frac{1}{k}\sum (x_i-\bar x)^2$ then the bias is $\mathbb E[s_k^2-\sigma^2] = \frac{n-1-k}{k}\sigma^2$ while the variance is $\mathrm{Var}( s_k^2) = \frac{2(n-1)}{k^2} \sigma^4$ and the expected square of the error is the variance plus the square of the bias, i.e. $\mathbb E[(s_k^2-\sigma^2)^2] = \frac{{n^2-2nk+k^2 +2k -1}}{k^2}\sigma^4$
It is common to consider $k={n-1},{n},{n+1}$
For predictive purposes it may not be that you want to minimise the variance of an estimator (if you do, then just choose a constant such as $0$) or that you want to eliminate the bias of an estimator as ends in themselves; it may be more that you really want to minimise their combined effect on the error.
One case is when you deal with high parametric case and use penalised estimators, in you question it could be logistic regression with lasso. The shrinking decreeses variance by killing some features (possibly significant), but at the same time it reduces the bias.
Another case which comes to my mind is consistent model selection (in regression setup, though, for example, with BIC): with probability going to one we choose correct model, though for a moderate data set the selected model can be "smaller", which could give a big bias.
