Gower Distance and PAM algorithm with Random Forest for Variable Selection

Question

I am currently working with cluster analysis and am trying to create clusters based on the important variables. My data consists of both categorical and continuous variables thus I have used the Gower distance measure to dissimilarities/similarities between observations, then I applied the PAM algorithm for clustering. I would like to know which variables are the best for my clusters I was going to use Random Forest but I have never used this technique before and would like to understand a bit better how it works and if it is the right technique to use. Below is the coding I have used in R:

#clearing all variables
rm(list=ls(all=TRUE))
set.seed(9850)

library(cluster) # for gower similarity and pam
library(dplyr)
library(ggplot2)
library(readr)
library(Rtsne) # for visualization in lower dimension
library(factoextra) #clustering algorithms and visualization
library(tidyverse) # for sampling


#Getting the data
df <- read.csv(file.choose())
names(df)
attach(df)

#selecting Reg.Cust only
RegularCust <- df[Type =="Regular",]
sample20 <- RegularCust %>% sample_frac(0.10) 

#selecting continuous and categorical vars
sample20vars <- data.frame(sample20$serv,sample20$equip,sample20$address,sample20$city,sample20$zones,sample20$compe,                             sample20$OfferPrice,sample20$Finalprice,sample20$P_number,sample20$TV,sample20$T1,                       sample20$T2,sample20$Down,sample20$Up)


gower_dist <- daisy(sample20vars, metric = "gower")

gower_mat <- as.matrix(gower_dist)

#Print Most Similar Clients
sample20vars[which(gower_mat == min(gower_mat[gower_mat != min(gower_mat)]), arr.ind = TRUE)[1, ], ]

#Print Most dissimilar clients
sample20vars[which(gower_mat == max(gower_mat[gower_mat != max(gower_mat)]), arr.ind = TRUE)[1, ], ]



#finding the most optimal number of clusters
sil_width <- c(NA)
for(i in 2:8){  
  pam_fit <- pam(gower_dist, diss = TRUE, k = i)  
  sil_width[i] <- pam_fit$silinfo$avg.width  
}
plot(1:8, sil_width,
     xlab = "Number of clusters",
     ylab = "Silhouette Width")
lines(1:8, sil_width)

#summary of each cluster

k <- 2

pam_fit <- pam(gower_dist, diss = TRUE, k)

pam_results <- sample20vars %>%
  mutate(cluster = pam_fit$clustering) %>%
  group_by(cluster) %>%
  do(the_summary = summary(.))
pam_results$the_summary

tsne_obj <- Rtsne(gower_dist, is_distance = TRUE)

tsne_data <- tsne_obj$Y %>%
  data.frame() %>%
  setNames(c("X", "Y")) %>%
  mutate(cluster = factor(pam_fit$clustering))

ggplot(aes(x = X, y = Y), data = tsne_data) +
  geom_point(aes(color = cluster))

I hope someone can help with this as I am not sure if 2 clusters is the right amount of clusters and I wanted to check which are the best variables before doing clustering.

Answer 1

I'm doing a similar analysis right now and I see you have run the sil_width routine. It will generate a plot you can use to find the optimal number of clusters. Depending on your data, it might be crystal clear or you might have to pick one based on other criteria if several cluster numbers are close.

As far as the best variable choice, the software has no idea why you are clustering. It could be to determine if one cluster is easy to work with and the other is hard (assuming 2 clusters), or it could just as easily be which cluster is closest to the nearest invisible alien. So you need subject matter experts to help you choose variables which relate to the question you want to answer. It's not clear to me that the clustering algorithm itself can help you choose the best variables to use.

Bill Bentley