13
$\begingroup$

I'm trying to predict the success or failure of students based on some features with a logistic regression model. To improve the performance of the model, I've already thought about splitting up the students into different groups based on obvious differences and building separate models for each group. But I think it might be difficult to identify these groups by examination, so I thought of splitting the students up by clustering on their features. Is this a common practice in building such models? Would you suggest that I break it down into obvious groups (for example, first term students vs. returning students) and then perform clustering on those groups, or cluster from the start?

To try to clarify:

What I mean is that I'm considering using a clustering algorithm to break my training set for the logistic regression into groups. I would then do separate logistic regressions for each of those groups. Then when using the logistic regression to predict the outcome for a student, I would choose which model to use based on which group they best fit into.

Perhaps I could do the same thing by including a group identifier, for example, a 1 if the student is returning and a 0 if not.

Now you've got me thinking about whether it might be advantageous to cluster the training data set and using their cluster label as a feature in the logistic regression, rather than building separate logistic regression models for each population.

If it's useful to include a group identifier for those who are returning students vs. new students, might it also be useful to expand the list of groups? Clustering seems like a natural way to do this.

I hope that's clear ...

$\endgroup$
1
  • $\begingroup$ I think I do not grasp how "clustering" and the logistic regression model would interact or affect one another. Could you explain the difference between "clustering" in this context and including a group identifier as an explanatory variable in the regression? $\endgroup$
    – whuber
    Jul 1, 2012 at 17:12

5 Answers 5

4
$\begingroup$

I believe that if you have a significant difference in your dependent variable between your clusters then the approach of clustering first will DEFINITELY be helpful. Regardless of your chosen learning algorithm.

It is my opinion that running a learnign algorithm on a whole base can cover up meaningful differences at a lower level of aggregation.

Anyone heard of simpson's paradox, it's a hard case of a deeper problem where you have different correlations in different groups that are covered up by larger sample noise and or weaker correlations of a larger group.

$\endgroup$
5
  • $\begingroup$ You may be right, but I don't follow your argument. Are you advocating that the OP run separate LRs on the found clusters, add the cluster index in addition to the covariates, or instead of the covariates? It's certainly true that covariates can be confounded w/ omitted variables in observational research, but are you saying that CA can generate information that isn't in the variables it's run on? As for Simpson's paradox, it is discussed on CV here if you're interested. $\endgroup$ Jul 2, 2012 at 3:49
  • $\begingroup$ I'm suggesting that the unsupervised analysis pull out homogenous groups with an arbitrary set of IV's (independent variables). Following this you can decide yourself if you progress with the same set of varbs or a new set or a combined set for the next stage of your modelling with the LR. The purpose is to build and tune 1 LR per cluster (given that the cluster's have significantly different DV values or frequencies). $\endgroup$
    – clancy
    Jul 2, 2012 at 5:54
  • $\begingroup$ I have actually carried this out myself in the context of a take up model for cross selling Life Insurance products and found improved prediction on 2 of the clusters that was being diluted by a 3rd cluster. $\endgroup$
    – clancy
    Jul 2, 2012 at 5:59
  • $\begingroup$ I wonder if the model may have needed a spline term. Could you include a simulation of some data, a basic fit, CA, & final (improved) fit w/ cluster indicator? I'd be interested in seeing this, and playing with it a little to understand what's going on. $\endgroup$ Jul 2, 2012 at 12:14
  • $\begingroup$ Hi Gung, I'd love to but can't find the time. I am heavily invested with family, work and improving my modelling skills.I am just beginning to work with MARS modelling and am not sure if this will satisfy the same desired outcome as the described cluster + LR ensemble. $\endgroup$
    – clancy
    Jul 2, 2012 at 22:46
9
$\begingroup$

Your proposed general approach - using latent partitions to assign different data points to different base classifiers - is a well-researched approach toward classification.

The reason these methods are not widely used is likely because they are relatively complicated and have longer running times than logistic regression or SVMs. In many cases, it seems that they can lead to better classification performance.

Here are some references:

  • Shahbaba, B. and Neal, R. "Nonlinear models using Dirichlet process mixtures"

  • Zhu, J. and Chen, N. and Xing, E.P. "Infinite Latent SVM for Classification and Multi-task Learning"

  • Rasmussen, C.E. and Ghahramani, Z. "Infinite mixtures of Gaussian process experts"

  • Meeds, E. and Osindero, S. "An alternative infinite mixture of Gaussian process experts"

$\endgroup$
1
$\begingroup$

When creating multi-segmented models, I think the best approach is to create segments that speak to real differences in the underlying distributions. First-term students vs returning students is a great example, as the predictor distributions will likely be very different for these two populations. More importantly, these differences have an intuitive explanation.

$\endgroup$
2
  • $\begingroup$ I get the value of an intuitive explanation -- it helps you interpret your model. But is there not reason to think that if you cluster people into groups based on their similarity, in terms of the features you have available, you'll get a similar benefit, albeit not with the same interpretability? I guess the idea behind the use of clustering is that when it comes to identifying groups which don't correspond neatly with categories we use in every day life, machines are better than humans ... $\endgroup$
    – dave
    Jun 30, 2012 at 22:44
  • $\begingroup$ And, additionally, that if you train a regression model on a set of similar students, that model will be more accurate in its predictions of those students' success than a model which was trained using a broader set of students. $\endgroup$
    – dave
    Jun 30, 2012 at 22:47
1
$\begingroup$

I want to acknowledge from the beginning that I know relatively little about clustering. However, I don't see the point of the procedure you describe. If you think, for example, that first term vs. returning students might be different, why not include a covariate that indexes that? Likewise if you think another feature of the students is relevant, you can include that as well. If you are worried that the relationship between your primary predictor of interest and the success rate might differ, you could also include the interaction between that predictor and first term vs. returning, etc. Logistic regression is well equipped to address these questions via including such terms in the model.

On the other hand, so long as you only cluster on these features, and do so first (without looking at the response), I don't see any problems arising. I suspect this approach would be inefficient, with each model having lower power because it is fit on only a subset of the data, but I don't think it would bias the parameters or invalidate the tests. So I suppose you could try this if you really want to.

Update:

My guess is that it would be best (i.e., most efficient) to fit one model with all the data. You could include some additional covariates (such as returning vs. not) beyond your primary interest, and a grouping indicator that you discovered via having run a cluster analysis beforehand. However, if the covariates that went into the cluster analysis are also made available to the logistic regression model, I'm not sure if I can see what would be gained over just including all of the covariates in the LR model without the cluster indicator. There may well be an advantage to this that I'm not familiar with, since I'm not expert in cluster analysis, but I don't know what it would be. It seems to me that the CA would not generate additional information that wasn't already there in the covariates, and thus wouldn't add anything to the LR model. You could try it; maybe I'm wrong. But my guess is that you would just burn a few extra degrees of freedom.

A different approach would be to enter the cluster indicator into the LR model instead of the covariates on which it's based. I doubt this would be beneficial. The CA won't be perfect, any more than any other analysis ever is, and so moving from the original covariates to the derived cluster indicator is likely to entail some amount of information loss. (Again, I don't know that, but I strongly suspect it's true.) Again, you could try it both ways and compare as an academic exercise, although just trying a lot of stuff and settling on the outcome that looks best is frowned upon if you want to take your results seriously.

I don't want to just carp on cluster analyses. There may be many benefits of them in general, and there may be a good use for them here. However, as I understand your situation, I think just building a LR model with the covariates you think might be relevant is the way to go.

$\endgroup$
1
$\begingroup$

If you are not tied to logistic regression I would suggest you use a random forest classifier because it has a sort of built in clustering. The idea would be to use use the proximity matrix to cluster. The proximity matrix is the N_Obs by N_Obs matrix for the fraction of out of bag trees where the observations where in the same terminal node. You can then aggregate this into a feature level by feature level matrix where the elements are the average of the fraction in the proximity matrix. You would then cluster all levels together when they past a threshold and see if this improves your prediction. It is likely best to take a step-wise iterative approach to find the optimal clustering but you can choose a threshold in other ways. When this clustering is done you could replace the feature with the cluster labels or add the cluster labels as a new feature. I suppose at this point you could switch back to logistic regression if you really wanted.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.