2
$\begingroup$

The features in data sometimes contains missing values, which mean the value is unknown. If I replace unknown value with a special normal value like "0", then the clustering algorithms will trade them as they are same, because both are 0. But "both unknown" doesn't mean they are "same", actually unknown means nothing.

Should I ignore all data which contains unknown feature, or ignore all features which contains unknown value? Both are too overkill I think.

Is there any better way to handle unknown data for clustering problem?

Thanks

$\endgroup$
2
$\begingroup$

If you exclude features with missing values, you might bias your conclusions or lose information.

Consider a dataset with 10 patients and their cholesterol values. You are interested in predicting cholesterol values based on these features. You might have one feature, age at beginning of study, and one feature # chol checks last month. The latter is missing in 5 of the patients because they were so healthy that they decided to not even follow up by sending you the data. In this case, if you exclude that feature, you might exclude your best predictor.

A better way is to note that all of those patients who didn't follow up also happened to be the young ones. Also you might note that for the 5 patients who did have # cholesterol check records sent to you, the data was like this

age # checks
50     10
60     20
70     30
80     40

You can see that there is a relationship between cholesterol checks and age; you could even figure out the parameters of a regression. You can use this regression to then fill in the missing values for the young patients. This is the idea behind matrix completion.

The values that you impute will however be single values, and you won't have a sense of how good they really are. For making predictions, you can hold out a test set and see whether your imputation method actually improve results. For clustering, depending on your application, because its difficult to evaluate your imputation method as a step in some larger pipeline, it might be wise to also consider multiple imputation as suggested by @mkt.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.