# Does preclustering help to build a better predictive model?

For the task of churn modelling I was considering:

1. Compute k clusters for the data
2. Build k models for each cluster individually.

The rationale for that is,that there is nothing to prove, that the population of subsribers is homogenous, so its reasonable to assume that data-generating process may be diffrent for diffrent "groups"

My question is, is it an appropriate method? Does it violate anything, or is it considered bad for some reason? If so, why?

If not, would you share some best practices on that issue? And 2nd thing - is it generally better or worse to do preclustering than model tree (As defined in Witten,Frank - classification/regression tree with models at the leafs. Intuitively it seems that decision-tree stage is just another form of clustering, but idk if it has any advantages over "normal" clustering.).

-

There is a method called clusterwise regression that solves similar problem (first clusters data and then builts predictive models). See for example this.

-
I looked it up here: tandfonline.com/doi/abs/10.1080/00273170701836653 and found following in the abstract: "n some cases, most of the variation in the response variable is explained by clustering the objects, with little additional benefit provided by the within-cluster regression models. Accordingly, there is tremendous potential for overfitting with clusterwise regression". Doesn't really seem promising. – Ziel Oct 12 '12 at 13:26
Ok, but they do not say that it always fails. I have never used that method, I only know that it may be combination of supervised and unsupervised approach but there is a small number of papers that use this method. – Miroslav Sabo Oct 12 '12 at 13:30
In addition, most applications that I found are about marketing and finance so maybe it is suitable especially for this kind of data. – Miroslav Sabo Oct 12 '12 at 13:39
It does seem very intuitive for the field of marketing - churn,cross/upsell. – Ziel Oct 12 '12 at 13:50

I'm dealing with similar problem these days. I have hundreds of feature to build classifier. After trying different models (ex: random forests, gradient boost, etc...), I still got low precision/recall. So I'm trying to do some clustering then build classifiers in different groups. My concern is, just like Anony-Mousse says, how can I gain more information from the classifier if I use all the information in clustering? So here's what I gonna do next:

1. Use some features (less, according to prior knowledge) to do clustering.
2. Use other features (more) to train classifiers.

I think it may also helps to reduce complexity, wish it helps.

-

Two points that are too long to be a comment:

• pure clusters (i.e. containing cases of one class only) are no problem per se: so called one-class classifiers model each class independent of all others. They can perfectly deal with this.

• However, if the data clusters in a way that the classes are quite separated, i.e. the clusters are rather pure, this means that a very strong structure exists, a structure that cluster analysis is able to find without guidance by the class labels. This means that certain types of classifiers such as nearest neighbour methods based on the same distance measure used by the cluster analysis are appropriate for the data.

• The other possibility, situations where the clusters are not pure, but a combination of cluster and classification methods can do well is appropriate for trees. The tree will do the part of the clustering (and pure nodes are not considered a problem.) Here's an artificial example, a 2 cluster version of the XOR-problem:

• another way to include the cluster information without running the risk of having pure clusters would be to use the clustering as a feature generation step: add the outcome of the cluster analysis as new variates to the data matrix.

• You ask whether it is bad for some reason: one pitfall is that this appoach leads to models with many degrees of freedom. You'll have to be particularly careful not to overfit.

• Have a look at model-based-trees, e.g. mbq's answer here I think they implement a concept that is very close to whar you look for. They can be implemented as forest as well: e.g. R package mobForest.

-

Well, if your clusters are really good, your classifiers will be crap. Because they have not enough diversion in their training data.

Say your clusters are perfect i.e. pure. You can't even properly train a classifier there anymore. Classifiers need positive and negative examples!

Random Forest are very successful in doing the exact opposite. They take a random sample of the data, train a classifier on that, and then use all of the trained classifiers.

What might work is to use clustering, and then train a classifier on every pair of clusters, at least if they disagree enough (if a class is split into two clusters, you still cannot train a classifier there!)

-
The purpose of the clustering is not to find "pure" clusters, i.e. ones that are awesome in discriminating my target variable. The purpose of the clustering is finding groups homogenous in the "other" area. To give an example: I think that in churn there are "quality-only" customers and "cost-optimizining" customers. I don't think I should assume that relevant features for classifiation are same in both groups so i want to build separate model for each group. Of course I don't have explicit "quality" and "cost" groups, hence the idea for clustering to derive such groups first from data. – Ziel Oct 12 '12 at 14:37
Any kind of extra imbalancedness and correlation in the data can harm. See, a classifier may want to discern "quality only" and "cost optimizing". If he only gets one group, he cannot make use of this distinction. – Anony-Mousse Oct 12 '12 at 15:12
Maybe I explained it poorly. "Quality only" and "cost optimizing" are latent, unobserveable. Maybe they dont even exist, its a hypothetis. For the sake of explanation let's say they do. Say that for QO group relevant variables for discrimination are X1-X5. For CO group relevant variables are X6-X10. It's no use throwing both groups in one classifier, because you dont have an observable dummy "QO vs. CO". You will get some average betas for X1-X10, not suitable for either group. So the idea is to do clustering and find relevant groups, that may be govern by diffrent data generating process. – Ziel Oct 12 '12 at 15:31
But only if you do a two-level approach, first classify by the clusters, then evaluate the cluster classifier. Otherwise, the constant classifier is useless. Then you are putting all the burden to the clustering. – Anony-Mousse Oct 15 '12 at 16:57
You can of course do this, but chances are that your clusters aren't that good, and that you are better off with a proper ensemble, of "overlapping" classifiers. Just like RandomForests does. – Anony-Mousse Oct 15 '12 at 23:35
show 8 more comments

Building $k$ clusters and then $k$ corresponding models is absolutely feasible. The pathologic case noted in the comments wherein the clusters perfectly separate the outcome variables would pose difficulties for classifiers is a theoretical problem, but one which I think is unlikely (especially in a high dimensional case). Furthermore, if you could build such clusters, you could then just use those clusters for prediction!

In addition, if the process begins with $N$ samples, the classifiers can only use $N/k$ samples. Thus, a more powerful approach would be to use the clusters in building a single classifier that incorporates the heterogeneity in the clusters using a mixture of regressions. In model-based clustering, one assumes the data are generated from a mixture distribution $Y_i \sim N(\mu_i, \sigma_i^2)$ where $i=1$ with probability $\pi$ and $i=2$ with probability $1-\pi$ and $\mu_1 \neq \ \mu_2$ and $\sigma_1^2 \neq \sigma_2^2$. A mixture regression is an extension that allows one to model the data as being dependent on co-variates; $\mu_i$ is replaced with $\beta_i X_i$, where the $\beta_i$ have to be estimated. While this example is for a univariate, Gaussian case, the framework can accommodate many data (multinomial-logit would be appropriate for categorical variables). The flexmix package for R provides a more detailed description and of course a relatively easy and extensible way to implement this approach.

Alternatively, in a discriminative setting, one could try incorporating cluster assignments (hard or soft) as a feature for training the classification algorithm of choice (e.g. NB, ANN, SVM, RF, etc.)

-