10
$\begingroup$

I'm doing multi-class classification on the Abalone dataset by divided the abalone into age groups young, adult and old.

While doing so, I found that the columns for the abalone size and weights were highly correlated. I'm also using the sex categories via one-hot encoding.

enter image description here

The dataset info also mentioned that "Data set samples are highly overlapped. Further information is required to separate completely using affine combinations."

What are the implications of this high correlations, and how do I perform my feature engineering knowing this?

Improve this question
$\endgroup$
5
  • 4
    $\begingroup$ A common approach for highly correlated features is to do dimension reduction. In the simplest case, this can be done via PCA, a linear technique. For your particular case, PCA might be reasonable, but you might want to do it on log-transformed features, due to allometric scaling (e.g. weight ~ length$^3$). $\endgroup$
    – GeoMatt22
    Commented Sep 13, 2016 at 2:28
  • 3
    $\begingroup$ It should be mentioned, that correlation is generally only a problem if you are doing inference on the parameters of your model. SInce you mentioned machine learning, I would assume you are doing prediction. Correlation between features does not generally affect the predictive accuracy of learning models. $\endgroup$
    – Matthew Drury
    Commented Sep 13, 2016 at 3:32
  • $\begingroup$ @GeoMatt22 thanks for the answer. i did do a PCA on this but didn't really know how to interpret it. adding to your comment, could it be that I would use something like a volume (diameter x length X height) instead of length^3? $\endgroup$
    – sfactor
    Commented Sep 13, 2016 at 4:42
  • $\begingroup$ @MatthewDrury thanks for the answer. yes I'm using this in a 3 class classifier model. could you elaborate what do you mean by "correlation is generally only a problem if you are doing inference on the parameters of your model". how would that be different than doing ML? $\endgroup$
    – sfactor
    Commented Sep 13, 2016 at 4:43
  • 1
    $\begingroup$ @sfactor you could do as you suggest and add $V=DLH$, and it could be this is correlated to weight $W=\rho V$ (i.e. $\rho=$density, not corr-coef). This makes sense in terms of dimensional analysis, yes? My point was that these types of (allometric-scaling) relationships will be log-linear, e.g. $\log W = \log\rho + \log D + \log L + \log H$. So rather than making $V$ a-priori, you could let PCA on the log-transformed features figure out what to do automatically. $\endgroup$
    – GeoMatt22
    Commented Sep 13, 2016 at 4:58

1 Answer 1

Reset to default
6
$\begingroup$

In theory, this should not affect your ability to make predictions - after all, the only truly useless data would be a restated column (or a column whose values can be directly derived from some other - e.g. having radius and circumference in two columns). Just because your features are correlated does not mean they are not useful, in fact, this correlation could be valuable if your dataset is in fact representative of what is out there "in the wild".

However, if your dataset is limited, then you may run into trouble, as highly correlated data will provide precious little extra information about the subject. PCA is a great candidate for this, as mentioned in the above comment. Random Forests are also promising, as they can inform you which columns play the biggest part in classifying your data. Gradient Boosting classifiers can also help with data that is resistant to classification by more elementary methods.

At any rate, I'm curious to hear what your baseline is with basic classifiers!

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ thanks for the answer. this data seems to be hard to classify, as I couldn't get more than 62% accuracy with my KNN, SVM and RF classifiers. i did try PCA (with the sex categories also as a feature besides the ones mentioned). using the first 4 PCAs (95% variance). there was no improvements with those either. looks like a dataset that is hard to do ML with. $\endgroup$
    – sfactor
    Commented Sep 13, 2016 at 4:46
  • $\begingroup$ Bummer, could be that it's not a great dataset for classification. But on the other hand those can be fun because they're challenging. Also out of curiosity: is there class imbalance as well or do you have balanced classes? $\endgroup$
    – Derek Janni
    Commented Sep 13, 2016 at 6:26
  • 1
    $\begingroup$ Well they are not perfectly balanced, more like a 40/35/25 ratio. I'm using Stratified KFold cross validation to make sure all my training and test datasets have the same ratio. $\endgroup$
    – sfactor
    Commented Sep 13, 2016 at 6:40
  • $\begingroup$ Interesting.. comment back if you come up with anything, I'm curious about the problem! $\endgroup$
    – Derek Janni
    Commented Sep 19, 2016 at 5:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.