Highly Correlated Datasets - Why and What Next? I've got three datasets (of different biological data) that are highly correlated - such that if I use the typical findCorrelation from the caret package with a cutoff of 0.99, then 5000 predictors are removed from one with 8000, another loses 648 from 654, and the last loses 645 from around 700.
My questions are these:

*

*How can I examine my data to understand why they could be highly correlated?


*Should I remove my data that is very highly correlated, even if it means the removal of nearly all data (a cutoff of 0.75 would remove everything, therefore 0.99 seems a better option)?


*And finally, if I'm subjecting my data to analysis with PLS or PCA methods do I even need to remove highly correlated features at all?
Hope someone can help out, thanks for reading!
 A: I don't suggest findCorrelation to get rid of highly correlated observations. I've just read the documentation by ?caret::findCorrelation and still don't know which correlation measurement method the function uses, but I think it uses pearson correlation.
Pearson correlation is not always a good way to measure correlation, specifically when you see enormous percentage (like %99 etc.) there should be something wrong. Let me try to explain with an example;
First, I created a dummy dataset with two different variable whose variables come from different distributions;
set.seed(1453)

dummy <- data.frame(a=rnorm(n = 100,mean = 50,sd = 100),
                    b=runif(min = 100,max = 500,n = 100))

Now I'm checking the corrlation between two variable with base cor function (which uses pearson correlation coefficient as default)
cor(dummy)

A matrix: 2 × 2 of type dbl
a   b
a   1.0000000   0.1844875
b   0.1844875   1.0000000

Pearson correlation coefficient says there is %18 positive correlation between two variables. Now, I am creating an outlier which isn't close the mean of both two distributions ;
hack_row <- data.frame(a=10000,b=10000)

dummy2 <- rbind(dummy,hack_row)

Now, Let me check the correlation again;
cor(dummy2)

    a   b
a   1.0000000   0.9896067
b   0.9896067   1.0000000

Pearson says %98 correlation between two variables ! Only one outlier can make it possible.
In short, some outliers can mislead us. %98 is not that realistic. We could use a non-parametric correlation measurement method like spearman correlation with the same data (with outlier).
cor(dummy2,method = 'spearman')

    a   b
a   1.0000000   0.1544787
b   0.1544787   1.0000000

Spearman says there is %15 positive correlation between two variables, which is way realistic than pearson correlation.
To get rid of highly correlated observations, you can use a non-parametric correlation measurement method and without findCorrelation function. Probably it will take a little more effort but you will no longer lose high amount of data.
You can continue with PCA analysis with your trusty uncorrelated data.

EDIT:
I've just learned that caret's findCorrelation takes a correlation matrix as an argument, changing cor's method argument to a non-parametric one fixes everything :)
