# How do I use a correlation matrix in Big Data?

I'm fairly new to Big Data and have been reading the book 'Applied Predictive Modeling' by Max Kuhn, Kjell Johnson. I'm trying to understand how to use the correlation matrix in the context of big data.

This is an example of a correlation matrix that one can generate in R:

https://cran.r-project.org/web/packages/corrplot/vignettes/corrplot-intro.html

In big data, the datasets are huge with hundreds of predictor variables so expect this square to also be huge.

I understand that to prevent multicollinearity, one should not take a extreme blue or the extreme red pair of predictor variables as they are correlated and this could affect the results coming out of your predictive model. Instead, you should pick pairs of variables with low correlation, like qsec and drat with a correlation of 0.09.

However, is even generating this matrix even relevant in the big data context as per my understanding, most predictive models have feature selection in place so the correlated values should be filtered out by the predictive model already saving us from doing so manually?

I can see the relevance of the correlation matrix for a small linear regression model if you want to see whether something is correlated or not so you make a decision on whether to take out the variable of the model or not, but I just cannot seem to wrap my head around the relevance of this matrix in the Big Data context.

• Correlation is usually for EDA. By the time you get to modeling you should have an idea of which variables will be useful. You wouldn't try to fit a model with an irrelevant and uncorrelated variable. – Jon Dec 7 '16 at 17:21
• Tools are tools. Some fit specific situations better than others but don't get distracted by words like "big data". It's mostly hyperbole. – Jon Dec 7 '16 at 17:22