# How should I deal with highly correlated features?

I'm doing multi-class classification on the Abalone dataset by divided the abalone into age groups young, adult and old.

While doing so, I found that the columns for the abalone size and weights were highly correlated. I'm also using the sex categories via one-hot encoding.

The dataset info also mentioned that "Data set samples are highly overlapped. Further information is required to separate completely using affine combinations."

What are the implications of this high correlations, and how do I perform my feature engineering knowing this?

• A common approach for highly correlated features is to do dimension reduction. In the simplest case, this can be done via PCA, a linear technique. For your particular case, PCA might be reasonable, but you might want to do it on log-transformed features, due to allometric scaling (e.g. weight ~ length$^3$). – GeoMatt22 Sep 13 '16 at 2:28
• It should be mentioned, that correlation is generally only a problem if you are doing inference on the parameters of your model. SInce you mentioned machine learning, I would assume you are doing prediction. Correlation between features does not generally affect the predictive accuracy of learning models. – Matthew Drury Sep 13 '16 at 3:32
• @GeoMatt22 thanks for the answer. i did do a PCA on this but didn't really know how to interpret it. adding to your comment, could it be that I would use something like a volume (diameter x length X height) instead of length^3? – sfactor Sep 13 '16 at 4:42
• @MatthewDrury thanks for the answer. yes I'm using this in a 3 class classifier model. could you elaborate what do you mean by "correlation is generally only a problem if you are doing inference on the parameters of your model". how would that be different than doing ML? – sfactor Sep 13 '16 at 4:43
• @sfactor you could do as you suggest and add $V=DLH$, and it could be this is correlated to weight $W=\rho V$ (i.e. $\rho=$density, not corr-coef). This makes sense in terms of dimensional analysis, yes? My point was that these types of (allometric-scaling) relationships will be log-linear, e.g. $\log W = \log\rho + \log D + \log L + \log H$. So rather than making $V$ a-priori, you could let PCA on the log-transformed features figure out what to do automatically. – GeoMatt22 Sep 13 '16 at 4:58