Consider the following bootstrapping scheme, common with weak classifiers: 0. Given training data N samples by P predictors
- sample the training data with replacement k times (for a large k)
- fit unbiased strong classifier, e.g. Bayesian regression (or lasso), to each of the k data frames
Note that in step 1, sampling with replacement, for a large N, results in (e-1/e)*N unique rows.
Now let's think through the interpretation of item 1 in terms of simulated annealing. In the context of likelihood-based methods, score looks like exp((logLik + penalty)/T). for large N, penalty is much smaller than logLik and we can treat it as a second-order effect. We can then re-write the score as exp((sum_1..N(logLik_i)/T). If N is large, then sum_1..N(logLik_i)/T is approximately the same as sum_(subset of 1/T elements from 1..N)(logLik_i). In other words, simulated annealing is approximately a data sub-sampling scheme.
Because the # of unique rows after sampling with replacement is roughly 2/3N, we are roughly training the classifier on that many rows. Increasing the number of non-unique rows is approximately the same as training below T=1 and down to T = 2/3 in our annealing analogy. In other words, for a strong classifier where the features are meaningful and not expected to cancel out, sampling with replacement should lead to significant overtraining.
I have written a fairly clunky and partially proprietary script that I can't share here where I tested this assertion and, indeed, the ensemble of classifiers performs a lot better when training data for each model is reduced to only unique rows.
- Does this make sense or does this sound crazy?
- Has anyone seen this written up? Could I ask for a link?
- If this hasn't been written up, what kind of journal would be good? Is Monte Carlo angle acceptable to the statistical audience or is there a better avenue of approach?