I just read this post and several other websites, but I still don't understand what bagging is.
I understand it is an algorithm for machine learning, that it improves stability and accuracy of the algorithm and decreased the variance of my prediction, but what is the main idea behind this algorithm? Is it to put a dataset into a bag?
For example, to me the main idea behind boosting is to boost records that are weighted incorrectly.
Bootstrapping is a concept in statistics of approximating the sampling distribution of a statistic by repeatedly sampling from a given sample of size $n$. We construct $B$ samples, each of size $n$, by sampling with replacement from the original sample. The statistic of interest is calculated for each of the $B$ samples. For sufficiently large $B$, we have a good idea of how the statistic is distributed. Roughly speaking, this distribution indicates the range of values of a statistic and how dense these values are.
Bagging, or Bootstrap AGGregatING, is an extension of bootstrapping to classification and regression problems. The main idea is to sample with replacement from the training data so that we now have $B$ training data sets, each having $n' \le n$ observations. The machine-learning algorithm is trained on each of the $B$ data sets to form a committee. When predicting (or classifying) future test observations, we ask each trained algorithm in the committee for its prediction. We then compute a (weighted) average of the $B$ predictions to obtain a single prediction.
The simplest approach is to weight each of the $B$ committee members equally. However, several variants are available that reduce the weight of less reliable committee members (e.g., poor classification accuracy, multiple outliers are present, etc).
