How to validate clusters after calculating Gower distances and Ward's clustering in R

Question

I am trying to apply Ward's clustering on a mixed types dataset, and wanna explain what I did (maybe helpful to others), and I have some questions regarding this analysis, mainly how to validate my clusters.

So, let me explain what I did in detail:

1. I started with a dataset containing 53 variables. These variables are either numerical or binary. The first variable contains the participant number, so will not be used in the clustering. I directly coded the categorical data into binary variables so that I didn't have to transform these variables.

2. Then I checked for normality of the variables for all numerical variables, using the Shapiro-Wilk test. Significance values below 0.05 will be log-transformed in the next step. Binary variables don't need a normality check.

3. I calculated distance using the daisy library. Here, I log-transformed the variables with a value lower than 0.05 in the second step with 'logratio', the other variables were appointed as asymmetric binary and symmetric binary. I read that Gower distance is an appropriate metric for mixed data types, and that's why I carried out this step this way.

gower_dist <- daisy(mydata[, -1], metric = "gower", type = list(logratio = c(4,5,6,7,8,9,10,11,12,13,17,24,25,26,44,45,49,50,51,52), asymm = c(14,15,16,18,19,20,21,23,27,28,29,30,31,32,34,35,36,37,38,39,40,41,42,43,46,47,48), symm = c(1,3)))

4. Then I used Ward's method for clustering, and I plotted a dendogram with the results.

hc1 <- hclust(gower_dist, method = "ward" )

plot(hc1) # display dendogram

This resulted in the following output:

I think that I am in the right direction with my analysis, but have some problems with finding the right number of clusters. I wanted to use pvclust() since it provides p-values for hierarchical clustering, and that is what I am interested in. However, it seems that this package is not usable when using Gower distances. Does anyone know another way to find p-values in R for Ward's clusters using the Gower distance?

Next to this, I have some small doubts on the correctness of my analysis. These are the following:

• I read that using binary variables can be problematic in https://www.researchgate.net/publication/223532418_Hierarchical_clustering_of_mixed_data_based_on_distance_hierarchy. I assumed that with a high dimensional data set, this would not be a large problem, is that true?
• According to https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4120293/ log transforming data does not help make data less variable or more normal and may, in some circumstances, make data more variable and more skewed. Is it still wise to apply log transformation to the variables? Also, I think that using Ward's method is wise to use for high dimensional datasets, is that true?
• I wanted to use 'Partial data cluster analysis' to deal with missing values in the data (as explained in https://www.displayr.com/5-ways-deal-missing-data-cluster-analysis/), but I am not sure how I dealt with missing data the way I carried out the analysis now. This also doesn't seem to be documented.

Answer 1

I am no expert, but I have been reading up a bit. I have read that binary/categorical variables will dominate the clustering when you use Gower (because they take the most extreme values, 0 and 1, while other variables are constrained to be between 0 and 1). It is possible to tweak Gower distance to use a different normalization and to deal with missing values, but that involves coding. In a high dimensional data set, the clustering can be mucked up by addition of irrelevant variables (I believe) so it can make sense to try to reduce the dimensionality. (One suggestion below. Others use PCA, I think, but that is going to be sensitive to outliers.) Regarding skewed data, I believe skewness is a big problem for most clustering, and I would advise using Verardi and Vermandele (Outlier identification for skewed and/or heavy-tailed unimodal multivariate distributions) to remove outliers, and then use Box-Cox to de-skew the data (if log doesn't work). (Not sure how to de-skew categorical variables though, hahaha.) Then you can use a clustering algorithm. Wang, Yabes and Chang (Hybrid Density- and Partition-based Clustering Algorithm for Data with Mixed-type variables) describes a way to go about clustering, including removal of variables, with mixed data. It is a multi-step process, and you could add the skewness step I mentioned above. I don't know about Ward's.