Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I have a question about cluster analysis. Normally when there is less than 10% missing data and its missing at random, then it can be ignored.
But how should I handle the missing data for a cluster analysis if it is higher than 10%? As far as I can see normal statistical methods cannot be used, because then the respondents with missing values will look like each other and then be grouped.
It depends on your similarity measure.
You can define similarity such that it considers missing data to be the same, to be different, to be different with a certain probabilitiy, ... whatever you need. Just don't forget to first work on your data, and distance, then on clustering.
Maybe these are just language issues but I think they are worth clarifying.
Firstly you state missing values with less than 10% missing are ignored. This is not completely true. If less than 10% are missing (and data is scarce and therefore precious), the first approach people employ is to try and impute missing values. If you drop away each input with even one missing value you might end up dropping a lot of the entries. Points are generally only dropped when a lot of their attributes are missing
You state "Normal Statistical Models cannot be used...". This is not completely true. A lot of the imputation techniques end up employing EM techniques with Normal update steps. In a lot of the techniques, missing values are given the same dummy label of (say -1). I would not do this in the case of clustering techniques because it has the possibility of messing up your distance calculations.
