Clustering and regression with high dimensional, mixed type data

I have been looking at several similar questions and answers discussing these issues but I cannot say there is a clear answer to what I am posting here. There seems to be a general confusion with the whole clustering process so I am trying to summarise the problems I have faced for anyone interested and potentially find a solution as well.

First of all, I will describe my data: I have a dataset of approximately 20000 observations, about 40 independent variables and 1 dependent. Out of the 40 variables, I have a few (around 6 or 7) that are categorical and most of the others are counts. What I am trying to do is to use all these variables to predict the dependent, by applying regressions with out of sample (80% training and 20% testing). So far, my errors are too big; hence I have thought of applying clustering to my dataset and then apply regressions within the clusters in order to successfully predict the dependent variable with the lowest error possible within each cluster.

The problems I am facing now are related more to methodological approaches rather than technical, although there are some technical issues as well. I will try to list them and elaborate respectively.

1) Dimensionality reduction: do I have to do it or is it optional? There is no clear answer to that. If I am about to apply a method for this what is going to be? Remember that my data is of mixed type. The most comprehensive discussion I found is here where my conclusion is that PCA cannot be applied to this type of data and other methods are suggested. To my understanding the only way to go through this is to apply Factor Analysis of Mixed Data (FAMD) which is also a function in R. Is this correct? are there any other ways to do this? If this is the only option I have a problem there in coding. What am I suppose to use for clustering of the FAMD outcomes?

a similar discussion can also be found here although this was even more confusing for me.

2) Clustering algorithm: Let's now assume that dimensionality reduction is not needed. What type of clustering to apply and how to deal with the categorical features? My findings so far end up in two conclusions. If partition clustering is used there is only one - namely Kprototypes - algorithm that can handle mixed data. I believe this is correct. One other approach is to use "gower distance" although this is kind of unclear to me. To my understanding, Gower distance is the only one that can be applied to mixed data (i.e. other distances such as Euclidean cannot handle categorical variables). But; is gower distance to be used only in hierarchical clustering?? People's suggestions are not very clear at this, so there is some confusion on how gower distance is used. Can someone clarify this? Maybe someone refer to partition and hierarcghical clustering and elaborate more on what to use at each? I have also read that algorithms such as Kmeans are not appropriate for count data, but no clear explanation on that as well... I am also not interested in one-hot encoding.

just in case anyone else is interested I have found similar links and discussions here, here and here, although these all led me to more questions than answers

3)Number of clusters: OK I know this is a big discussion, but again I am summarising here. There are several metrics discussed here such as the Calinsky criterion, Dunn's index, the "elbow" method, BIC and so on. For mixed type of data though, the only one I have managed to make it work is the "elbow" method iterating with the kprototypes algorithm. Is that correct?

4) Data transformation and scaling: Again I am unsure of when and if this is something to apply when clustering. To my understanding it is advisable when variables are of different units and ranges. But what type of transformation/scaling to use? Should it be standardisation, normalisation, log transformation when the distributions are skewed? What happens when you have mixed categorical and count data like in my case? I have been through a lot of published papers that do not normalise the data. Can someone please elaborate on this?

5) In general what is the correct process to follow in my problem? I believe it is something like Data scaling->dim.reduction->number of clusters->Clustering->regression.

Any help on any of these issues would be appreciated. I am coding in R so any coding examples would also be appreciated.

migrated from stackoverflow.comFeb 7 at 6:45

This question came from our site for professional and enthusiast programmers.

This has way too many questions, enough to write a whole manuscript, you need to do more research. I will only give you some basic information:

1. Two options: dimensionality reduction (PCA, FA, ...) or feature selection, I would recommend the second approach, since it is much easier to interpret and comes built-in some models
2. Gower distance should be fine, it can be used in all clustering algorithms, not just hierarchical
3. Elbow method or GAP statistic is fine
4. Normalization, if done, would be done before Gower, standardization is fine
5. normalization -> feature selection -> gower -> clustering -> number of clusters

Regression after clustering does not make sense.

Also, there is a clustering algorithm that was made with exactly high-dimensional data in mind, PAM (Partitioning Around Medoids or Nearest Shrunken Centroids), which performs a lot of these steps for you. Have a look at the pamr package.

• user2974951 Thank you for the answer. I am aware that this is a long question covering many issues around clustering. What I am trying to do is summarise all the information I gathered from several sources including this site as well as text books and papers from Hastie, Bishop and others. It seems that there isn't something that comprehensively covers the questions presented here in a tangible and logical manner and flow. Any references/sources would be much appreciated. – Alex Feb 7 at 15:46
• @Alex "R in Action, Data Analysis and graphics with R" by Robert I. Kabacoff. – user2974951 Feb 8 at 7:38