let's say that we have a simple, binary classification problem (with many predictors and many observations) and want to fit for example some kind of boosting algorithm to obtain resutls. Let's also assume that we are able to identify two clusters with some kind of clustering algorithm. My concern is whether it is possible to obtain better prediction results when boosting algorithm is fitted separately for each cluster, not on the whole dataset? Theoretically, clustering is performed on the same variables as predictive model, but maybe there is some kind of logic in such a solution?

What is more, maybe it is worth trying different model for the other cluster and this might gain better results?

  • $\begingroup$ Clustering is more difficult than classification. And you then need to map new data first to the clusters, then classify. Just stacking more things usually makes things worse and less predictable, and also less reliable/stable. $\endgroup$ – Has QUIT--Anony-Mousse Jun 24 '18 at 7:42
  • $\begingroup$ And what do you think about another (similar but not the same) approach to find clusters within given class (for example minority class, so called '1') and then apply predictive model separately, with majority class not clustered? Would that make sens? $\endgroup$ – MarkSt Jun 29 '18 at 18:17
  • $\begingroup$ Well, clustering still is much harder than classification. But if one class clusters extremely well, splitting it in "class A variant 1" and "class A variant 2" could improve things. I've seen this hypothesis before, but have just not seen any actually success story where the results improved significantly compared to, say, random forests or SVM. $\endgroup$ – Has QUIT--Anony-Mousse Jun 29 '18 at 21:50
  • $\begingroup$ Thanks. And what do you think about using clustering as an undersampling method for majority class in case of imbalanced data? For me it seems resonable and from what I have read so far it might enhance the results of classification for example when used with boosting. $\endgroup$ – MarkSt Jun 30 '18 at 12:21
  • $\begingroup$ Upsampling the rare class seems more promising. Or using methods that don't require this. $\endgroup$ – Has QUIT--Anony-Mousse Jun 30 '18 at 16:31

For me it makes sense using clustering as a method of feature engineering.
Assuming your intuition on the relation between predictors clusters and target variable is correct, helping models finding possibly complicated relations between predictors(by using a clustering algorithm) is a solid step.
In practice I would use the clustering results as features(cluster number- categorical) for the boosting algorithm.

Your second question - "is it a good idea training a different model for each cluster". I cant think of a yes/no answer for this, but when I face these situations I try deciding based on the amount of shared relevant(to the problem) information between clusters. If I believe this shared information is far less than the noise shared I would consider separating models training.

Another way tackling this issue is by using hierarchical models.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.