Incremental improvement for boosting By adding additional factors, will the fitting result of a boosting algo (say Ada boosting) guaranteed to be improved?
From my experiment, adding additional factors could make the prediction accuracy worse. But I don't understand how can the optimization get worse since you can always set a zero weight on the factors you don't like.
In addition, is there any method that solves this issue so additional factors would make prediction result worse (let's not worry about over fitting for now.)
 A: I would say, that You (as a human) can set (manually set) zero weight on those factors, but machine learning algorithm will not necessarily do so.
One problem with additional factors is overfitting. (I think, no further explanation is needed here)
To illustrate another problem let us consider a decision tree. As you move down the tree, you get fewer and fewer data points for each node. With some probability it may happen that the irrelevant variable will be chosen for the split. Since this is irrelevant variable, this split is actually wrong and worsens your solution. The fewer data points you have at node, the higher is the probability that this will happen. (I learned about this problem from the book by Witten, Frank, Hall "Data Mining". I tried to find a link now, by without success.)
One more problem is that additional factors may be correlated with the existing ones, thus causing multicollinearity
These problems are general and not specific for boosting (and also not solved by boosting)
A: In CART, and in tree-based algorithms such as Boosted trees and Random Forests, using irrelevant predictors is not an issue, because such variables are largely ignored during the tree-growing process, providing you do have some strong predictors in your data set (this is the way the CART algorithm works). See for instance this paper. In fact, it was even shown (Elements of Statistical Learning, section 15.3.4, p. 596) that with 6 relevant and 100 noise variables, the probability of selecting a relevant variable at any split was still 0.46. This means that the noise variables are mostly discarded.
