Selecting optimal iteration for GBM purposed to obtain propensity score estimation I am interested in propensity score estimation by GBM. I am reading Propensity Score Analysis by Bai and Pan. The book suggested several metric to evaluate balancing of propensity score by GBM. In GBM estimation, one needs to choose iteration which determines the over-fitting. However, the book does not distinguish between training data and testing data. Normally, I would have all three parts training, validation and testing data. Suppose I am using standardized bias for ATE to evaluate balancing.
Q: Should standardized bias be applied to test data to obtain optimal iteration? I believe this is the case as GBM fits training data well already.
 A: The use of GBM for propensity score analysis is a bit different from its usual use, which is to make predictions without overfitting. In propensity score analysis, the purpose of GBM is to estimate propensity scores that, when conditioned on (e.g., through weighting), yield covariate balance. Every unit needs a propensity score, so you have to make predictions for the entire sample.
Here's how GBM works for propensity score analysis:

*

*Train a GBM on the entire sample up to a user-supplied number of trees

*Generate in-sample predictions from the model at each tree

*Use the predictions as propensity scores and use them to compute weights for each unit in the sample

*Compute a balance statistic at each tree as a function of the weights

*Use the weights that yield the best balance in the final analysis to estimate the treatment effect

Every step uses the full sample; there is no splitting into training, testing, and validation sets. This doesn't lead to overfitting because the goal is not to have good predictions; it's to find good balance. A model that perfectly predicts treatment status will not yield good balance, so there is protection from overfitting.
This methodology is described in McCaffrey et al. (2004), which is the original reference for GBM for use in propensity score analysis.
There has been debate about the best balance measure to use to choose the number of trees. The answer depends on unknowable features of the true outcome model. Only the energy distance as described by Huling and Mak (2022) has statistical properties that guarantee bias reduction when the balance statistic is minimized, which is why it is implemented in WeightIt for use with GBM. Some other (a bit outdated) research on this topic comes from Parast et al. (2017).

Huling, J. D., & Mak, S. (2022). Energy Balancing of Covariate Distributions (arXiv:2004.13962). arXiv. https://doi.org/10.48550/arXiv.2004.13962
McCaffrey, D. F., Ridgeway, G., & Morral, A. R. (2004). Propensity Score Estimation With Boosted Regression for Evaluating Causal Effects in Observational Studies. Psychological Methods, 9(4), 403–425. https://doi.org/10.1037/1082-989X.9.4.403
Parast, L., McCaffrey, D. F., Burgette, L. F., de la Guardia, F. H., Golinelli, D., Miles, J. N. V., & Griffin, B. A. (2017). Optimizing variance-bias trade-off in the TWANG package for estimation of propensity scores. Health Services and Outcomes Research Methodology, 17(3), 175–197. https://doi.org/10.1007/s10742-016-0168-2
