I'm fairly new to Big Data and have been reading the book 'Applied Predictive Modeling' by Max Kuhn, Kjell Johnson. I'm trying to understand how to use the correlation matrix in the context of big data.

This is an example of a correlation matrix that one can generate in R:


enter image description here

In big data, the datasets are huge with hundreds of predictor variables so expect this square to also be huge.

I understand that to prevent multicollinearity, one should not take a extreme blue or the extreme red pair of predictor variables as they are correlated and this could affect the results coming out of your predictive model. Instead, you should pick pairs of variables with low correlation, like qsec and drat with a correlation of 0.09.

However, is even generating this matrix even relevant in the big data context as per my understanding, most predictive models have feature selection in place so the correlated values should be filtered out by the predictive model already saving us from doing so manually?

I can see the relevance of the correlation matrix for a small linear regression model if you want to see whether something is correlated or not so you make a decision on whether to take out the variable of the model or not, but I just cannot seem to wrap my head around the relevance of this matrix in the Big Data context.

  • $\begingroup$ Correlation is usually for EDA. By the time you get to modeling you should have an idea of which variables will be useful. You wouldn't try to fit a model with an irrelevant and uncorrelated variable. $\endgroup$ – Jon Dec 7 '16 at 17:21
  • $\begingroup$ Tools are tools. Some fit specific situations better than others but don't get distracted by words like "big data". It's mostly hyperbole. $\endgroup$ – Jon Dec 7 '16 at 17:22

The sensitivity of your model to highly correlated values will depend on the model selection. Random forest models handle correlated variables pretty well, but that doesn't mean you are really benefiting from having them there.

A random forest model may pick any one of the correlated variables as a predictor without a substantial preference of one over the other. Once one is used, the importance of the other correlated variables goes down. This could affect your interpretation of the data later on if you aren't aware of the other correlations. You could incorrectly assume one of the correlated variables is more important than the other for the model.

The correlation matrix is useful for exploratory data analysis. Once you decide what kind of model you want to build, you can decide whether filtering highly correlated variables is appropriate for the situation. The caret package in R has nice,simple tools for data pre-processing.

  • $\begingroup$ I think you put it nicely that it is for EDA purposes so that you can decide which model would work best with the data. I guess you would eye ball the matrix and make a decision on whether the dataset is highly correlated or not? In the example matrix diagram I posed in my question, I would assume from an eyeball view that it is highly correlated given the few occurrences of lightly coloured numbers and higher occurrences of darkly coloured blues and reds? $\endgroup$ – Simon Dec 7 '16 at 19:48
  • $\begingroup$ There is certainly a lot of correlation but depending on the data, that happens a lot. I work a lot with shape data, so it makes sense that a lot of features are correlated (area, perimeter, major axis length, etc). I generally use a threshold of 0.85 (abs. value) for filtering my variables. But as with all thresholds, it is somewhat arbitrary. $\endgroup$ – Doc Yall Dec 7 '16 at 22:54

Where you have lots of possible predictors, some highly correlated, one possibility is to combine them into a smaller number of independent components using PCA, ICA or similar methods.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.