I'm new in data analysis area and didn't have very strong statistical background...

Now, I'm trying to filter out those numeric columns which have high correlation. At this moment, I don't plan to use dimensional reduction algorithms such as PCA because I'm still collecting data, and there are too many columns, I hope to remove some columns first before linking different sources of data together.

Playing with PCA and machine learning models will be after linking those sources of data.

So, at this "before" stage, I'm trying to use R library caret, findCorrelation() method, but I could not find how does this method work.

Now my question is, I should use this method after data cleaning (such as, deal with missing data, outliers, data imbalance) or it doesn't matter to do it before or after?

  • $\begingroup$ I would say that you should do it afterwards; that's what I typically do. $\endgroup$ – topepo Sep 29 '16 at 16:28
  • 2
    $\begingroup$ How do you distinguish your procedure from dimensional reduction algorithms? Isn't that exactly what you're trying to do? $\endgroup$ – whuber Sep 29 '16 at 17:38
  • $\begingroup$ If I link all the columns from different data sources together and use dimensional reduction, that will be too much columns, and it's very difficult to do this link (there will be too many joins and it may generate many duplicated values when do the join...). Therefore, now I'm trying to filter out unnecessary columns from each data sources, finally link the rest together, and put to dimensional reduction. $\endgroup$ – Cherry Wu Sep 29 '16 at 17:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.