Skip to main content
Cross Validated

Return to Answer

edited body
Source Link
stans
stans
  • 1.8k
  • 8
  • 10

I agree with @ziggystar. As the number of bootstrap samples $k$ converges to infinity, bagged estimate of the linear model converges to the OLS (Ordinary Least Squares) estimate of the linear model run on the whole sample. The way to prove this is seeing that bootstrap "pretends" that the population distribution is the same as the empirical distribution. As you sample more and more data sets from this empirical distribution, the average of estimated hyperplanes will converge to the "true hyperplane" (which is OLS estimate run on the whole data) by asymptotic properties of Ordinary Least Squares.

Also, bagging is not always a good thing. Not only does it not fight the bias, it may increase the bias in some peculiar cases. Example: $$ X_1, X_2, ..., X_n \sim Be(p) $$ (Bernoulli trials which take value 1 with probability $p$ and value 0 with probability $1-p$.). Further, let us define parameter $$ \theta = 1_{\{p > 0\}} $$ and try to estimate it. Naturally, it suffices to see a single data point $X_i = 1$ to know that $\theta = 1$. The whole sample may contain such a data point and allow us to estimate $\theta$ without any error. On the other hand, any bootstrap sample may not contain such a data point and lead us to wrongly estimating $\theta$ with 0 (we adopt no Bayesian framework here, juts good old method of maximum likelihood). In other words, $$ {\rm Bias}_{\rm\ bagging} = {\rm Prob(in\ a\ bootstrap\ sample\ X_{(1)} = ... = X_{(n)} = 0)} > 0, $$ conditional on $\theta = 1$.

I agree with @ziggystar. As the number of bootstrap samples $k$ converges to infinity, bagged estimate of the linear model converges to the OLS (Ordinary Least Squares) estimate of the linear model run on the whole sample. The way to prove this is seeing that bootstrap "pretends" that the population distribution is the same as the empirical distribution. As you sample more and more data sets from this empirical distribution, the average of estimated hyperplanes will converge to the "true hyperplane" (which is OLS estimate run on the whole data) by asymptotic properties of Ordinary Least Squares.

Also, bagging is not always a good thing. Not only does it not fight the bias, it may increase the bias in some peculiar cases. Example: $$ X_1, X_2, ..., X_n \sim Be(p) $$ (Bernoulli trials which take value 1 with probability $p$ and value 0 with probability $1-p$.) Further, let us define parameter $$ \theta = 1_{\{p > 0\}} $$ and try to estimate it. Naturally, it suffices to see a single data point $X_i = 1$ to know that $\theta = 1$. The whole sample may contain such a data point and allow us to estimate $\theta$ without any error. On the other hand, any bootstrap sample may not contain such a data point and lead us to wrongly estimating $\theta$ with 0 (we adopt no Bayesian framework here, juts good old method of maximum likelihood). In other words, $$ {\rm Bias}_{\rm\ bagging} = {\rm Prob(in\ a\ bootstrap\ sample\ X_{(1)} = ... = X_{(n)} = 0)} > 0, $$ conditional on $\theta = 1$.

I agree with @ziggystar. As the number of bootstrap samples $k$ converges to infinity, bagged estimate of the linear model converges to the OLS (Ordinary Least Squares) estimate of the linear model run on the whole sample. The way to prove this is seeing that bootstrap "pretends" that the population distribution is the same as the empirical distribution. As you sample more and more data sets from this empirical distribution, the average of estimated hyperplanes will converge to the "true hyperplane" (which is OLS estimate run on the whole data) by asymptotic properties of Ordinary Least Squares.

Also, bagging is not always a good thing. Not only does it not fight the bias, it may increase the bias in some peculiar cases. Example: $$ X_1, X_2, ..., X_n \sim Be(p) $$ (Bernoulli trials which take value 1 with probability $p$ and value 0 with probability $1-p$). Further, let us define parameter $$ \theta = 1_{\{p > 0\}} $$ and try to estimate it. Naturally, it suffices to see a single data point $X_i = 1$ to know that $\theta = 1$. The whole sample may contain such a data point and allow us to estimate $\theta$ without any error. On the other hand, any bootstrap sample may not contain such a data point and lead us to wrongly estimating $\theta$ with 0 (we adopt no Bayesian framework here, juts good old method of maximum likelihood). In other words, $$ {\rm Bias}_{\rm\ bagging} = {\rm Prob(in\ a\ bootstrap\ sample\ X_{(1)} = ... = X_{(n)} = 0)} > 0, $$ conditional on $\theta = 1$.

Source Link
stans
stans
  • 1.8k
  • 8
  • 10

I agree with @ziggystar. As the number of bootstrap samples $k$ converges to infinity, bagged estimate of the linear model converges to the OLS (Ordinary Least Squares) estimate of the linear model run on the whole sample. The way to prove this is seeing that bootstrap "pretends" that the population distribution is the same as the empirical distribution. As you sample more and more data sets from this empirical distribution, the average of estimated hyperplanes will converge to the "true hyperplane" (which is OLS estimate run on the whole data) by asymptotic properties of Ordinary Least Squares.

Also, bagging is not always a good thing. Not only does it not fight the bias, it may increase the bias in some peculiar cases. Example: $$ X_1, X_2, ..., X_n \sim Be(p) $$ (Bernoulli trials which take value 1 with probability $p$ and value 0 with probability $1-p$.) Further, let us define parameter $$ \theta = 1_{\{p > 0\}} $$ and try to estimate it. Naturally, it suffices to see a single data point $X_i = 1$ to know that $\theta = 1$. The whole sample may contain such a data point and allow us to estimate $\theta$ without any error. On the other hand, any bootstrap sample may not contain such a data point and lead us to wrongly estimating $\theta$ with 0 (we adopt no Bayesian framework here, juts good old method of maximum likelihood). In other words, $$ {\rm Bias}_{\rm\ bagging} = {\rm Prob(in\ a\ bootstrap\ sample\ X_{(1)} = ... = X_{(n)} = 0)} > 0, $$ conditional on $\theta = 1$.