A common decomposition of the error incurred when forming a predictive model is into three pieces.
1) Bayes Error: Even the best predictor will sometimes be wrong. Imagine predicting height based on gender. If you had the best predictor available you would still incur error because height does not depend solely on gender. The best predictor is typically called the Bayes predictor.
2) Approximation Error: When forming predictive models, because we want a tractable problem, and because we do not want to over-fit to the data (see 3), we restrict our set of models to some family. For example, in ordinary least squares regression we typically restrict ourselves to a linear model with normal noise which has fixed variance. If the nature of the data generating mechanism does not follow these rules, even the best predictor in this family to which we've restricted ourselves will have more error than the Bayes predictor.
3) Estimation Error: Once we've restricted ourselves to some family of predictors, we must use our data to pick one predictor from that family. What if we do not choose the right one? Then we incur more error. To be clear, I am not referring to picking the wrong predictor by accident, but rather by statistical inference on a limited set of data.
One of the most fundamental issues in machine learning is the interplay between approximation error and estimation error. As we enlarge our family of predictors our approximation error monotonically decreases, as we are able to capture more complex relationships. However, as our family of predictors increases, our estimation error increases as we over-fit more.
An extreme example of this is fitting a polynomial model to scalar data $x_i$, $y_i$. Imagine that the data was generated by some cubic polynomial plus error. Now suppose we increase the maximum degree in our polynomial $d = 0, 1, 2, 3, \dots$.
The first prediction will be the sample mean, lots of approximation error, little estimation error (but still some as this will most likely not be the true mean). As we increase $d$ we are trading approximation error for estimation error.
Once we hit $d=3$ we will probably make our best predictions, after all, we have all of the flexibility needed to produce the Bayes predictor, so we are only limited by the limited size of our data.
Eventually, as we increase $d$ we will have no error on our training set at all, and our data will have cooked up some higher order relations that don't exist at all.