Gradient boosting machine accuracy decreases as number of iterations increases I'm experimenting with the gradient boosting machine algorithm via the caret package in R.
Using a small college admissions dataset, I ran the following code:
library(caret)

### Load admissions dataset. ###
mydata <- read.csv("http://www.ats.ucla.edu/stat/data/binary.csv")

### Create yes/no levels for admission. ### 
mydata$admit_factor[mydata$admit==0] <- "no"
mydata$admit_factor[mydata$admit==1] <- "yes"             

### Gradient boosting machine algorithm. ###
set.seed(123)
fitControl <- trainControl(method = 'cv', number = 5, summaryFunction=defaultSummary)
grid <- expand.grid(n.trees = seq(5000,1000000,5000), interaction.depth = 2, shrinkage = .001, n.minobsinnode = 20)
fit.gbm <- train(as.factor(admit_factor) ~ . - admit, data=mydata, method = 'gbm', trControl=fitControl, tuneGrid=grid, metric='Accuracy')
plot(fit.gbm)

and found to my surprise that the model's cross-validation accuracy decreased rather than increased as the number of boosting iterations increased, reaching a minimum accuracy of about .59 at ~450,000 iterations.

Did I incorrectly implement the GBM algorithm? 
EDIT:
Following Underminer's suggestion, I've rerun the above caret code but focused on running 100 to 5,000 boosting iterations:
set.seed(123)
fitControl <- trainControl(method = 'cv', number = 5, summaryFunction=defaultSummary)
grid <- expand.grid(n.trees = seq(100,5000,100), interaction.depth = 2, shrinkage = .001, n.minobsinnode = 20)
fit.gbm <- train(as.factor(admit_factor) ~ . - admit, data=mydata, method = 'gbm', trControl=fitControl, tuneGrid=grid, metric='Accuracy')
plot(fit.gbm)

The resulting plot shows that the accuracy actually peaks at nearly .705 at ~1,800 iterations:

What's curious is that the accuracy didn't plateau at ~.70 but instead declined following 5,000 iterations.
 A: Codes to reproduce a similar result, without grid search,
mod = gbm(admit ~ .,
      data = mydata[,-5],
      n.trees=100000,
      shrinkage=0.001,
      interaction.depth=2,
      n.minobsinnode=10,
      cv.folds=5,
      verbose=TRUE,
      n.cores=2)

best.iter <- gbm.perf(mod, method="OOB", plot.it=TRUE, oobag.curve=TRUE, overlay=TRUE)
print(best.iter)
[1] 1487
pred = as.integer(predict(mod, newdata=mydata[,-5], n.trees=best.iter) > 0)
y = mydata[,1]
sum(pred == y)/length(y)
[1] 0.7225

A: The gbm package has a function to estimate the optimal # of iterations (= # of trees, or # of basis functions),
gbm.perf(mod, method="OOB", plot.it=TRUE, oobag=TRUE, overlay=TRUE)

You don't need caret's train for that.
A: What you have displayed is a classic example of overfitting. The small uptick in error comes from poorer performance on the validation portion of your cross-validated data set. More iterations should nearly always improve the error on the training set, but the opposite is true for the validation/test set. 
A: In general, boosting error can increase with the number of iterations, specifically when the data is noisy (e.g. mislabeled cases).  I wouldn't be able to say if this is your problem without knowing more about your data
Basically, boosting can 'focus' on correctly predicting cases that contain misinformation, and in the process, deteriorate the average performance on other cases that are more substantive.
This link (Boosting and Noise) shows a better description than I can provide of the issue.  
This paper (Random Classification Noise) by Long and Servedio provides more technical details of the issue.
