I have been looking at several similar questions and answers discussing these issues but I cannot say there is a clear answer to what I am posting here. There seems to be a general confusion with the whole clustering process so I am trying to summarise the problems I have faced for anyone interested and potentially find a solution as well.
First of all, I will describe my data: I have a dataset of approximately 20000 observations, about 40 independent variables and 1 dependent. Out of the 40 variables, I have a few (around 6 or 7) that are categorical and most of the others are counts. What I am trying to do is to use all these variables to predict the dependent, by applying regressions with out of sample (80% training and 20% testing). So far, my errors are too big; hence I have thought of applying clustering to my dataset and then apply regressions within the clusters in order to successfully predict the dependent variable with the lowest error possible within each cluster.
The problems I am facing now are related more to methodological approaches rather than technical, although there are some technical issues as well. I will try to list them and elaborate respectively.
1) Dimensionality reduction: do I have to do it or is it optional? There is no clear answer to that. If I am about to apply a method for this what is going to be? Remember that my data is of mixed type. The most comprehensive discussion I found is here where my conclusion is that PCA cannot be applied to this type of data and other methods are suggested. To my understanding the only way to go through this is to apply Factor Analysis of Mixed Data (FAMD) which is also a function in R. Is this correct? are there any other ways to do this? If this is the only option I have a problem there in coding. What am I suppose to use for clustering of the FAMD outcomes?
a similar discussion can also be found here although this was even more confusing for me.
2) Clustering algorithm: Let's now assume that dimensionality reduction is not needed. What type of clustering to apply and how to deal with the categorical features? My findings so far end up in two conclusions. If partition clustering is used there is only one - namely Kprototypes - algorithm that can handle mixed data. I believe this is correct. One other approach is to use "gower distance" although this is kind of unclear to me. To my understanding, Gower distance is the only one that can be applied to mixed data (i.e. other distances such as Euclidean cannot handle categorical variables). But; is gower distance to be used only in hierarchical clustering?? People's suggestions are not very clear at this, so there is some confusion on how gower distance is used. Can someone clarify this? Maybe someone refer to partition and hierarcghical clustering and elaborate more on what to use at each? I have also read that algorithms such as Kmeans are not appropriate for count data, but no clear explanation on that as well... I am also not interested in one-hot encoding.
3)Number of clusters: OK I know this is a big discussion, but again I am summarising here. There are several metrics discussed here such as the Calinsky criterion, Dunn's index, the "elbow" method, BIC and so on. For mixed type of data though, the only one I have managed to make it work is the "elbow" method iterating with the kprototypes algorithm. Is that correct?
4) Data transformation and scaling: Again I am unsure of when and if this is something to apply when clustering. To my understanding it is advisable when variables are of different units and ranges. But what type of transformation/scaling to use? Should it be standardisation, normalisation, log transformation when the distributions are skewed? What happens when you have mixed categorical and count data like in my case? I have been through a lot of published papers that do not normalise the data. Can someone please elaborate on this?
5) In general what is the correct process to follow in my problem? I believe it is something like Data scaling->dim.reduction->number of clusters->Clustering->regression.
Any help on any of these issues would be appreciated. I am coding in R so any coding examples would also be appreciated.