1
$\begingroup$

let's say that we have a simple, binary classification problem (with many predictors and many observations) and want to fit for example some kind of boosting algorithm to obtain resutls. Let's also assume that we are able to identify two clusters with some kind of clustering algorithm. My concern is whether it is possible to obtain better prediction results when boosting algorithm is fitted separately for each cluster, not on the whole dataset? Theoretically, clustering is performed on the same variables as predictive model, but maybe there is some kind of logic in such a solution?

What is more, maybe it is worth trying different model for the other cluster and this might gain better results?

$\endgroup$
5
  • $\begingroup$ Clustering is more difficult than classification. And you then need to map new data first to the clusters, then classify. Just stacking more things usually makes things worse and less predictable, and also less reliable/stable. $\endgroup$ Commented Jun 24, 2018 at 7:42
  • $\begingroup$ And what do you think about another (similar but not the same) approach to find clusters within given class (for example minority class, so called '1') and then apply predictive model separately, with majority class not clustered? Would that make sens? $\endgroup$
    – MarkSt
    Commented Jun 29, 2018 at 18:17
  • $\begingroup$ Well, clustering still is much harder than classification. But if one class clusters extremely well, splitting it in "class A variant 1" and "class A variant 2" could improve things. I've seen this hypothesis before, but have just not seen any actually success story where the results improved significantly compared to, say, random forests or SVM. $\endgroup$ Commented Jun 29, 2018 at 21:50
  • $\begingroup$ Thanks. And what do you think about using clustering as an undersampling method for majority class in case of imbalanced data? For me it seems resonable and from what I have read so far it might enhance the results of classification for example when used with boosting. $\endgroup$
    – MarkSt
    Commented Jun 30, 2018 at 12:21
  • $\begingroup$ Upsampling the rare class seems more promising. Or using methods that don't require this. $\endgroup$ Commented Jun 30, 2018 at 16:31

2 Answers 2

0
$\begingroup$

For me it makes sense using clustering as a method of feature engineering.
Assuming your intuition on the relation between predictors clusters and target variable is correct, helping models finding possibly complicated relations between predictors(by using a clustering algorithm) is a solid step.
In practice I would use the clustering results as features(cluster number- categorical) for the boosting algorithm.

Your second question - "is it a good idea training a different model for each cluster". I cant think of a yes/no answer for this, but when I face these situations I try deciding based on the amount of shared relevant(to the problem) information between clusters. If I believe this shared information is far less than the noise shared I would consider separating models training.

Another way tackling this issue is by using hierarchical models.

$\endgroup$
0
$\begingroup$

As has been pointed out, using clustering as a feature engineering step is common practice. If you think you have discreet clusters present in your data and those meaningfully contribute to your classification then you can use an appropriate clustering algorithm and use the output as a new feature and as such make a mini ensemble model. Although be careful when doing so, different clustering algorithms will produce very different results. It really depends on the question you are trying to answer, one popular application is anomaly detection.

As for constructing two models on separate clusters. Usually this approach is used when you know you have separate clusters of data, let's say you work in a bank and one group of transactions are outgoing transactions, the other incoming ones. In such cases it makes sense to make two separate models based on the knowledge you have about the data, no clustering needs to be used. Sure you could also just use a clustering algorithm and then make separate models for each cluster, but it's easier then to just build a gradient boosted tree model with the whole dataset and make it heavily depend on the feature that you got from clustering for its decision making, not that i would recommend it though. A much more robust way of doing it would be to make multiple features with different clustering algorithms (used in a appropriate way) and then let the gradient boosting model decide which features it wants to use.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.