I need to do some clustering with my data set. I have 200 attributes and 18 tuples only. So I am trying to do some data cleaning. I deleted all attributes that has 0 as data and reached till 165. Now for further data cleaning I am trying to use correlation. I created a correlation matrix, and deleting the attributes that as correlation coefficient of greater than 0.9. Is it a good method? Do I need to consider anything else as well.


Correlation does not imply the attributes are bad.

It is never a good idea to remove attributes just because of some number. Rather use these to guide you to understand your data.

I.e. look at highly correlated attributes: are they correlated by cause (e.g. two nearby temperature sensors will be highly correlated because they measure the same thing - you should then probably merge these attributes) or is the correlation unexpectedly but related to your problem? (E.g., shoe size and gender are supposedly correlated, but you probably shouldn't remove them).

It won't get you anywhere closer to solving your task if you blindly follow a recipe to remove attributes.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.