I want to find 4-10 clusters in survey data with 100 questions answered by 2000 individuals using a technique such as K-means or Gaussian Mixture Models. There is no response variable so the clustering technique needs to be unsupervised. The outcome of the clustering highly depends on the way the data is represented and preprocessed, causing me to identify multiple potential issues.
The responses to the questions can be either ordinal with
M possible values or categorical with
N possible values.
M will be either 3 or 10, whereas
N can be anything in the interval [2,8]. All questions have been answered by all users.
Below are the issues that I would like to get input on:
How should find the similarity between responses when I have these different kinds of variables? I would assume that the best way to input the ordinal information would be as a single variable/dimension, whereas a categorical variable will be represented using
Nvariables/dimensions. Should I scale the variables in a particular way to ensure equal weight to each question or is there some similarity measure I could use that fits this kind of data?
The questions may be subgrouped categories such as "Sociodemographic Information", "Communication Assessment", "Product Assessment" and "Brand Assessment". Suppose that "Communication Assessment" contains 80% of the questions, but we would still like to get a clustering result that uses equal amount of information from each category. How should I preprocess the data to get this result?
The surveys are sent out to a representative sample of the population. However, the responses are biased such that males are overrepresented and females are underrepresented. The difference is statistically significant. How should I include this information in the clustering process? Should I oversample users to get a representative dataset across gender, age, job type? The number of duplicate users created this way would probably be 10-15% of the total.
You should feel free to address other issues with clustering survey data that I have not identified.