In general, boosting has potentially a huge problem with overfitting. Nothing is easier than to end up with bad overfitting when using boosting.
However, by letting each model in the ensemble try to improve what the previous ones still get wrong, what boosting provides is a very flexible modeling approach that can fit quite complex data often pretty well. So, the main challenge is how to make this as flexible as needed, but to regularize the whole process enough to avoid overfitting. Depending on your boosting algorithm, the various regularization options provide a number of knobs one can turn to try to get that trade-off right (as assessed by e.g. some form of cross-validation).
What you say about more "clever" averaging of the different models in the ensemble may be something one could do, but that gets complicated fast (especially for boosting instead of parallel building of independent predictors like in random forest). Additionally, if there's lots of models in your ensemble that all contribute not that differently, simple averages/sums over lots and lots of them is often a pretty good idea. I guess the other thing to say is that many modern boosting algorithms like XGBoost and LightGBM have options for sampling a fraction of records and a fraction of predictors to use in building each model (or even for each split in a tree), these options are pretty helpful, because they address what you mention regarding addressing different training examples.