4
$\begingroup$

I'm looking for a good tutorial about bias/variance tradeoff. In particular, I'd like to find someone that explains how different algorithms in machine learning play in this tradeoff, and possibly how other things (feature selection/extraction, scaling, etc...) may affect this trade-off.

$\endgroup$
1
$\begingroup$

It depends exactly what you'd like to know about bias/variance, but a good theoretical description of the problem is in Andrew Ng's CS229 Lecture Notes.

In general, this tradeoff is about model complexity - how many parameters are you trying to estimate, and how "free/independent" are they. This can be formalized using learning theory, as in Andrew's notes, but just the general intuition is probably enough for practical applications.

To answer your specific examples: feature selection should decrease variance (since we will end up fitting fewer parameters), hopefully without increasing bias too much (if done well). I'm not sure what you mean by "scaling", but that's probably an implementation-level detail that won't be too related to the more abstract bias/variance issues.

$\endgroup$
1
$\begingroup$

I think the article Choosing Prediction Over Explanation in Psychology: Lessons From Machine Learning by Tal Yarkoni and Jacob Westfall will give you some insights into the bias-variance trade-off: http://jakewestfall.org/publications/Yarkoni_Westfall_choosing_prediction.pdf.

$\endgroup$
1
$\begingroup$

A nice and intuitive explanation is found in this book. It accompanies the ISLR book with more code and simulations.

The author is using simulation to show the decomposition of the MSE and also shows graphically the trade off as model complexity increases

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.