10
$\begingroup$

For the task of churn modelling I was considering:

  1. Compute k clusters for the data
  2. Build k models for each cluster individually.

The rationale for that is, that there is nothing to prove, that the population of subscribers is homogenous, so its reasonable to assume that data-generating process may be different for different "groups".

My question is, is it an appropriate method? Does it violate anything, or is it considered bad for some reason? If so, why?

If not, would you share some best practices on that issue? And another question: is it generally better or worse to do pre-clustering than model tree (As defined in Witten, Frank - classification/regression tree with models at the leafs. Intuitively it seems that decision-tree stage is just another form of clustering, but I don't know whether it has any advantages over "normal" clustering.).

$\endgroup$

5 Answers 5

3
$\begingroup$

There is a method called clusterwise regression that solves similar problem (first clusters data and then builts predictive models). See for example this.

$\endgroup$
4
  • 1
    $\begingroup$ I looked it up here: tandfonline.com/doi/abs/10.1080/00273170701836653 and found following in the abstract: "n some cases, most of the variation in the response variable is explained by clustering the objects, with little additional benefit provided by the within-cluster regression models. Accordingly, there is tremendous potential for overfitting with clusterwise regression". Doesn't really seem promising. $\endgroup$
    – Ziel
    Oct 12, 2012 at 13:26
  • $\begingroup$ Ok, but they do not say that it always fails. I have never used that method, I only know that it may be combination of supervised and unsupervised approach but there is a small number of papers that use this method. $\endgroup$
    – sitems
    Oct 12, 2012 at 13:30
  • $\begingroup$ In addition, most applications that I found are about marketing and finance so maybe it is suitable especially for this kind of data. $\endgroup$
    – sitems
    Oct 12, 2012 at 13:39
  • 1
    $\begingroup$ It does seem very intuitive for the field of marketing - churn,cross/upsell. $\endgroup$
    – Ziel
    Oct 12, 2012 at 13:50
3
$\begingroup$

Two points that are too long to be a comment:

  • pure clusters (i.e. containing cases of one class only) are no problem per se: so called one-class classifiers model each class independent of all others. They can perfectly deal with this.

  • However, if the data clusters in a way that the classes are quite separated, i.e. the clusters are rather pure, this means that a very strong structure exists, a structure that cluster analysis is able to find without guidance by the class labels. This means that certain types of classifiers such as nearest neighbour methods based on the same distance measure used by the cluster analysis are appropriate for the data.

  • The other possibility, situations where the clusters are not pure, but a combination of cluster and classification methods can do well is appropriate for trees. The tree will do the part of the clustering (and pure nodes are not considered a problem.) Here's an artificial example, a 2 cluster version of the XOR-problem:
    XOR cluster

  • another way to include the cluster information without running the risk of having pure clusters would be to use the clustering as a feature generation step: add the outcome of the cluster analysis as new variates to the data matrix.

  • You ask whether it is bad for some reason: one pitfall is that this appoach leads to models with many degrees of freedom. You'll have to be particularly careful not to overfit.

  • Have a look at model-based-trees, e.g. mbq's answer here I think they implement a concept that is very close to whar you look for. They can be implemented as forest as well: e.g. R package mobForest.

$\endgroup$
1
$\begingroup$

I'm dealing with similar problem these days. I have hundreds of feature to build classifier. After trying different models (ex: random forests, gradient boost, etc...), I still got low precision/recall. So I'm trying to do some clustering then build classifiers in different groups. My concern is, just like Anony-Mousse says, how can I gain more information from the classifier if I use all the information in clustering? So here's what I gonna do next:

  1. Use some features (less, according to prior knowledge) to do clustering.
  2. Use other features (more) to train classifiers.

I think it may also helps to reduce complexity, wish it helps.

$\endgroup$
1
$\begingroup$

Building $k$ clusters and then $k$ corresponding models is absolutely feasible. The pathologic case noted in the comments wherein the clusters perfectly separate the outcome variables would pose difficulties for classifiers is a theoretical problem, but one which I think is unlikely (especially in a high dimensional case). Furthermore, if you could build such clusters, you could then just use those clusters for prediction!

In addition, if the process begins with $N$ samples, the classifiers can only use $N/k$ samples. Thus, a more powerful approach would be to use the clusters in building a single classifier that incorporates the heterogeneity in the clusters using a mixture of regressions. In model-based clustering, one assumes the data are generated from a mixture distribution $Y_i \sim N(\mu_i, \sigma_i^2)$ where $i=1$ with probability $\pi$ and $i=2$ with probability $1-\pi$ and $\mu_1 \neq \ \mu_2$ and $\sigma_1^2 \neq \sigma_2^2$. A mixture regression is an extension that allows one to model the data as being dependent on co-variates; $\mu_i$ is replaced with $\beta_i X_i$, where the $\beta_i$ have to be estimated. While this example is for a univariate, Gaussian case, the framework can accommodate many data (multinomial-logit would be appropriate for categorical variables). The flexmix package for R provides a more detailed description and of course a relatively easy and extensible way to implement this approach.

Alternatively, in a discriminative setting, one could try incorporating cluster assignments (hard or soft) as a feature for training the classification algorithm of choice (e.g. NB, ANN, SVM, RF, etc.)

$\endgroup$
0
$\begingroup$

Well, if your clusters are really good, your classifiers will be crap. Because they have not enough diversion in their training data.

Say your clusters are perfect i.e. pure. You can't even properly train a classifier there anymore. Classifiers need positive and negative examples!

Random Forest are very successful in doing the exact opposite. They take a random sample of the data, train a classifier on that, and then use all of the trained classifiers.

What might work is to use clustering, and then train a classifier on every pair of clusters, at least if they disagree enough (if a class is split into two clusters, you still cannot train a classifier there!)

$\endgroup$
13
  • $\begingroup$ The purpose of the clustering is not to find "pure" clusters, i.e. ones that are awesome in discriminating my target variable. The purpose of the clustering is finding groups homogenous in the "other" area. To give an example: I think that in churn there are "quality-only" customers and "cost-optimizining" customers. I don't think I should assume that relevant features for classifiation are same in both groups so i want to build separate model for each group. Of course I don't have explicit "quality" and "cost" groups, hence the idea for clustering to derive such groups first from data. $\endgroup$
    – Ziel
    Oct 12, 2012 at 14:37
  • $\begingroup$ Any kind of extra imbalancedness and correlation in the data can harm. See, a classifier may want to discern "quality only" and "cost optimizing". If he only gets one group, he cannot make use of this distinction. $\endgroup$ Oct 12, 2012 at 15:12
  • 1
    $\begingroup$ But only if you do a two-level approach, first classify by the clusters, then evaluate the cluster classifier. Otherwise, the constant classifier is useless. Then you are putting all the burden to the clustering. $\endgroup$ Oct 15, 2012 at 16:57
  • 1
    $\begingroup$ Well, that's how I understood the OP. $\endgroup$ Oct 15, 2012 at 22:32
  • 1
    $\begingroup$ You can of course do this, but chances are that your clusters aren't that good, and that you are better off with a proper ensemble, of "overlapping" classifiers. Just like RandomForests does. $\endgroup$ Oct 15, 2012 at 23:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.