I need to build segmentation on a large customer dataset with more than 300K records and many variables, including continuous like income and age, ordinal like education level and membership level, nominal variables like occupation and race, binary like gender. I use SAS to perform this task and someone suggest me to do the following? 1. artificially assign numeric values to categorical variables if they have order in some sense, otherwise transform to binary (0,1) 2. proc varclus to reduce variables 3. proc fastclus to generate many small clusters and reduce data size 4. proc cluster for hierarchical clustering based on output files from last step, and determine optimal number of clusters 5. proc tree for final segment result
Is this a valid method? BesidesI have following questions:
I would like to know methods to identify and remove outliers other than excluding extreme or abnormal cases from univariate analysis. I also wonder how should I treat missing values in cluster analysis.(at present, I will fill in missing data with means if the missing percentage is not that high, and will stop using variables with lots of missing)
For variable selection, my understanding on proc varclus/factor is they only work with numeric variables. Although categorical variables can be transformed into numeric, yet it seems not that sensible from a statistical perspective. what are appropriate ways to select categorical variables?
proc fastclus deal with large dataset well, but I wonder it is suitable for my cases since there are mixed types of data. How can I deal with this large data size? is it appropriate for me to just transform categorical to numeric variables and use them in this procedure?
proc cluster can use distance matrix other than Euclidean. Therefore, I may use proc distance to calculate Gower's distance which work with all level of measurement as input for clustering. However, it is infeasible for such large dataset. Is that mean this method can not be used on big data? Any idea to solve this problems?
Since Wald method should not use Gower's distance, then what are suitable methods for Gower's distance
It will be great if someone can suggest me any appropriate approaches to my segmentation work. thank you.