Cluster analysis with boosting models for better predictions?

Question

let's say that we have a simple, binary classification problem (with many predictors and many observations) and want to fit for example some kind of boosting algorithm to obtain resutls. Let's also assume that we are able to identify two clusters with some kind of clustering algorithm. My concern is whether it is possible to obtain better prediction results when boosting algorithm is fitted separately for each cluster, not on the whole dataset? Theoretically, clustering is performed on the same variables as predictive model, but maybe there is some kind of logic in such a solution?

What is more, maybe it is worth trying different model for the other cluster and this might gain better results?

Answer 1

For me it makes sense using clustering as a method of feature engineering.
Assuming your intuition on the relation between predictors clusters and target variable is correct, helping models finding possibly complicated relations between predictors(by using a clustering algorithm) is a solid step.
In practice I would use the clustering results as features(cluster number- categorical) for the boosting algorithm.

Your second question - "is it a good idea training a different model for each cluster". I cant think of a yes/no answer for this, but when I face these situations I try deciding based on the amount of shared relevant(to the problem) information between clusters. If I believe this shared information is far less than the noise shared I would consider separating models training.

Another way tackling this issue is by using hierarchical models.

Answer 2

As has been pointed out, using clustering as a feature engineering step is common practice. If you think you have discreet clusters present in your data and those meaningfully contribute to your classification then you can use an appropriate clustering algorithm and use the output as a new feature and as such make a mini ensemble model. Although be careful when doing so, different clustering algorithms will produce very different results. It really depends on the question you are trying to answer, one popular application is anomaly detection.

As for constructing two models on separate clusters. Usually this approach is used when you know you have separate clusters of data, let's say you work in a bank and one group of transactions are outgoing transactions, the other incoming ones. In such cases it makes sense to make two separate models based on the knowledge you have about the data, no clustering needs to be used. Sure you could also just use a clustering algorithm and then make separate models for each cluster, but it's easier then to just build a gradient boosted tree model with the whole dataset and make it heavily depend on the feature that you got from clustering for its decision making, not that i would recommend it though. A much more robust way of doing it would be to make multiple features with different clustering algorithms (used in a appropriate way) and then let the gradient boosting model decide which features it wants to use.