Clustering as a means of splitting up data for logistic regression

Question

I'm trying to predict the success or failure of students based on some features with a logistic regression model. To improve the performance of the model, I've already thought about splitting up the students into different groups based on obvious differences and building separate models for each group. But I think it might be difficult to identify these groups by examination, so I thought of splitting the students up by clustering on their features. Is this a common practice in building such models? Would you suggest that I break it down into obvious groups (for example, first term students vs. returning students) and then perform clustering on those groups, or cluster from the start?

To try to clarify:

What I mean is that I'm considering using a clustering algorithm to break my training set for the logistic regression into groups. I would then do separate logistic regressions for each of those groups. Then when using the logistic regression to predict the outcome for a student, I would choose which model to use based on which group they best fit into.

Perhaps I could do the same thing by including a group identifier, for example, a 1 if the student is returning and a 0 if not.

Now you've got me thinking about whether it might be advantageous to cluster the training data set and using their cluster label as a feature in the logistic regression, rather than building separate logistic regression models for each population.

If it's useful to include a group identifier for those who are returning students vs. new students, might it also be useful to expand the list of groups? Clustering seems like a natural way to do this.

I hope that's clear ...

Answer 1

I believe that if you have a significant difference in your dependent variable between your clusters then the approach of clustering first will DEFINITELY be helpful. Regardless of your chosen learning algorithm.

It is my opinion that running a learnign algorithm on a whole base can cover up meaningful differences at a lower level of aggregation.

Anyone heard of simpson's paradox, it's a hard case of a deeper problem where you have different correlations in different groups that are covered up by larger sample noise and or weaker correlations of a larger group.

Answer 2

Your proposed general approach - using latent partitions to assign different data points to different base classifiers - is a well-researched approach toward classification.

The reason these methods are not widely used is likely because they are relatively complicated and have longer running times than logistic regression or SVMs. In many cases, it seems that they can lead to better classification performance.

Here are some references:

• Shahbaba, B. and Neal, R. "Nonlinear models using Dirichlet process mixtures"

• Zhu, J. and Chen, N. and Xing, E.P. "Infinite Latent SVM for Classification and Multi-task Learning"

• Rasmussen, C.E. and Ghahramani, Z. "Infinite mixtures of Gaussian process experts"

• Meeds, E. and Osindero, S. "An alternative infinite mixture of Gaussian process experts"

Answer 3

When creating multi-segmented models, I think the best approach is to create segments that speak to real differences in the underlying distributions. First-term students vs returning students is a great example, as the predictor distributions will likely be very different for these two populations. More importantly, these differences have an intuitive explanation.

Answer 4

I want to acknowledge from the beginning that I know relatively little about clustering. However, I don't see the point of the procedure you describe. If you think, for example, that first term vs. returning students might be different, why not include a covariate that indexes that? Likewise if you think another feature of the students is relevant, you can include that as well. If you are worried that the relationship between your primary predictor of interest and the success rate might differ, you could also include the interaction between that predictor and first term vs. returning, etc. Logistic regression is well equipped to address these questions via including such terms in the model.

On the other hand, so long as you only cluster on these features, and do so first (without looking at the response), I don't see any problems arising. I suspect this approach would be inefficient, with each model having lower power because it is fit on only a subset of the data, but I don't think it would bias the parameters or invalidate the tests. So I suppose you could try this if you really want to.

Update:

My guess is that it would be best (i.e., most efficient) to fit one model with all the data. You could include some additional covariates (such as returning vs. not) beyond your primary interest, and a grouping indicator that you discovered via having run a cluster analysis beforehand. However, if the covariates that went into the cluster analysis are also made available to the logistic regression model, I'm not sure if I can see what would be gained over just including all of the covariates in the LR model without the cluster indicator. There may well be an advantage to this that I'm not familiar with, since I'm not expert in cluster analysis, but I don't know what it would be. It seems to me that the CA would not generate additional information that wasn't already there in the covariates, and thus wouldn't add anything to the LR model. You could try it; maybe I'm wrong. But my guess is that you would just burn a few extra degrees of freedom.

A different approach would be to enter the cluster indicator into the LR model instead of the covariates on which it's based. I doubt this would be beneficial. The CA won't be perfect, any more than any other analysis ever is, and so moving from the original covariates to the derived cluster indicator is likely to entail some amount of information loss. (Again, I don't know that, but I strongly suspect it's true.) Again, you could try it both ways and compare as an academic exercise, although just trying a lot of stuff and settling on the outcome that looks best is frowned upon if you want to take your results seriously.

I don't want to just carp on cluster analyses. There may be many benefits of them in general, and there may be a good use for them here. However, as I understand your situation, I think just building a LR model with the covariates you think might be relevant is the way to go.

Answer 5

If you are not tied to logistic regression I would suggest you use a random forest classifier because it has a sort of built in clustering. The idea would be to use use the proximity matrix to cluster. The proximity matrix is the N_Obs by N_Obs matrix for the fraction of out of bag trees where the observations where in the same terminal node. You can then aggregate this into a feature level by feature level matrix where the elements are the average of the fraction in the proximity matrix. You would then cluster all levels together when they past a threshold and see if this improves your prediction. It is likely best to take a step-wise iterative approach to find the optimal clustering but you can choose a threshold in other ways. When this clustering is done you could replace the feature with the cluster labels or add the cluster labels as a new feature. I suppose at this point you could switch back to logistic regression if you really wanted.