Average Error vs. Aggregate Error

Question

I was reading this paper on the history of Bagging Estimators (https://www.stat.berkeley.edu/~breiman/bagging.pdf) and came across the following section:

• I am having difficulty understanding the difference between "Phi_A" and "Phi_B" - especially in understanding how the term "Phi_A" suddenly appears when you expand the "average prediction error" (epsilon).

In this paper, it mentions that Phi_A refers to the "Aggregate Predictor" and Phi_B refers to the "Bagged Predictor". Until now, I always used to use these terms ("Aggregate" and "Bagging") interchangeably. For example, a Random Forest model can be thought of as aggregating Decision Trees on bootstrapped samples from the same dataset.

This being said, the next point is particularly confusing to me : The inequality shows us that that the "average prediction error" (epsilon) must be greater than or equal to the error from the "aggregated predictor error". Thus, this brings me to my question:

What is the difference between the "averaged prediction error", the "aggregated predictor error" and the "bagged error"? Thanks!

Answer 1

$\varphi_A$ in this excerpt refers to an estimator constructed by aggregating estimators trained on different datasets drawn from the distribution $P$. So that would be like training different models each on their own dataset and combining their predictions. The result is called the aggregated predictor. The aggregated prediction error is the error of the aggregated predictor on new data. We expect this error to be lower than the error of one individual model. The average of the individual models' prediction errors is the averaged prediction error.

With bagging, each predictor is not actually trained on its own independent dataset, so we don't get the aggregated predictor. Instead, predictors are trained on bootstrap random samples from just one dataset. In the excerpt, the distribution of bootstrap random samples is denoted as $P_{\mathcal{L}}$. The error of this model is the bagged prediction error.