1
$\begingroup$

I am working on a speech emotion recognition problem and my training dataset consists of about $4000$ points of $138$ features each. The highest (Pearson) correlation among the features is $0.3$ and there are only $7$ features which are correlated in the range $(0.3, 0.4)$ to the target values.

Does it make sense to investigate feature selection techniques in this case ? To my understanding it is not, since the correlations between the features and between the features and the target are quite low. However, I would appreciate your thoughts in this because I do not have much experience in this field. Thank you.

$\endgroup$
4
  • $\begingroup$ Why do you have to select fewer than $138$ of you features? Why not take all of them? Frank Harrell, for instance, has argued that feature selection is unstable (check the feature importance in a training set vs a holdout set) and that parsimony is the enemy of predictive ability. // Do you mean that you have a time series ("signal") where each observation (e.g., speaker) consists of $4000$ samples of $138$ variables? $\endgroup$
    – Dave
    Jan 25, 2022 at 22:30
  • $\begingroup$ The 4000 speech samples (observarions) are distributed to many different speakers.. Each speech sample is described by a 138-dim vector $\endgroup$
    – vggls
    Jan 26, 2022 at 10:58
  • $\begingroup$ So why not use all $138$ variables? $\endgroup$
    – Dave
    Jan 26, 2022 at 11:00
  • $\begingroup$ Currently I am using all of them.. I am just asking if spending time on investigating feature selection might yield better results.. $\endgroup$
    – vggls
    Jan 26, 2022 at 11:03

1 Answer 1

1
$\begingroup$

Since you are working on a speech emotion recognition problem, I assume that's quite complex data, and I assume you are not using simple linear methods like linear regression. Please correct me if I'm wrong in this assumption.

General Notes about your problem:

  1. Don't forget Pearson correlation is only taking into account linear correlation between variables. There might be non-linear (polynomial, logarithmic etc.) relationships between your variables. There could also be step-function-like relationships. All of these things would be poorly captured by Pearson correlation.
  2. Since your Pearson correlations are low, it seems that the relationships in the dataset (if any) very well might be non-linear and complex (especially given the subject of the dataset). If there are complex or non-obvious relationships to be discovered, then it might be the case that 4,000 data points isn't enough. It's a good amount, and it might be enough (depending on your model), but just keep in mind that it certainly isn't a huge amount of data by any stretch, especially given how many features you have. Think about it this way - your model will have to try to identify relationships between all 138 features vs. target variable, and it's only given 4,000 data points to do so. It might not be able to capture everything there is to capture, so it might make sense to whittle down your feature set.

That's a perfect set up to directly answer your question:

Yes, it absolutely makes sense to investigate feature selection techniques.

Reasons:

  1. Just because the Pearson correlation is low doesn't necessarily mean there's no relationship. Feature selection methods might help you quickly figure out whether there is any more complex relationship to be discovered.
  2. For the reason mentioned in the second note I wrote above, if there are unhelpful variables within your 138, depending on your choice of model, it might be very helpful to get rid of them so your model can focus on analyzing the relationships between the actually useful variables vs. target variable.

Pointers to get started on feature selection:

Again, it depends on your model, but broadly speaking, I would heavily recommend some version of Permutation Feature Importance to figure out which features are helpful. Read more here: https://scikit-learn.org/stable/modules/permutation_importance.html

There are various packages that implement it, like sklearn in Python and Boruta in R.

Quick tip for Permutation Feature Importance: In order to have a faster and more logical way of running this, try clustered Permutation Feature Importance (https://scikit-learn.org/stable/auto_examples/inspection/plot_permutation_importance_multicollinear.html#sphx-glr-auto-examples-inspection-plot-permutation-importance-multicollinear-py) . Essentially, group your 138 features into several groups (by which variables are most similar), and then run permutation feat. imp. on each of the entire groups, not on individual variables.

If that's a bit too complicated advanced, more simple feature selection methods include things like forward stepwise selection (add variables one at a time), backward stepwise selection (remove variables one a time) and LASSO regression (type of regression that simultaneously finds a model and removes obviously bad variables. 138 might be too many to feed right into it, though). These are all relatively straightforward to implement, and a peruse through Google should give you a good intuition/code for how to do all of them.

Side note: There are certain algorithms (like RandomForest) that often do not benefit greatly from feature selection. So, if are familiar with that technique and don't want to fuss with feature selection, that could be an option as well.

I hope this gives you good reasoning for why feature selection might be helpful even when correlation between variables and target is low, as well as some guidance about how you can get started running some feature selection on your data.

$\endgroup$
6
  • $\begingroup$ Thank you for the above.. Allow me a quick clarification.. The 4000 data come from 90 speakers.. If I use sklearn's built-in stepwise selection or permutation fetaure importance functions, the train/validation sets formed by the built-in functions in order to select features will not be speaker-independent.. Is this correct ? $\endgroup$
    – vggls
    Jan 26, 2022 at 11:29
  • $\begingroup$ Fyi you are correct that 4000 samples of 138-dim each is not a good balance for such a problem.. Eventually I had to ignore some emotion classes in order to get "better" results.. The 4K size refers to a 3-class dataset and still the performance is very mediocre.. $\endgroup$
    – vggls
    Jan 26, 2022 at 11:33
  • $\begingroup$ @john_b Why wouldn't it be speaker-independent? 90 speakers just refers to the data source, right? That is, you got your 4000 rows from 90 speakers. But is this really directly relevant to the analysis process? $\endgroup$ Jan 26, 2022 at 15:08
  • $\begingroup$ Ideally I would like to have different speakers on the training and the validating sets.. to avoid a classifier overfitted on the speaker.. For example in scikit-learn.org/stable/modules/generated/… there is a "cv" parameter.. So when the function splits the 4K dataset into k folds.. it will not take into account the speakers as well.. (no?).. it will just randomly select k subsets of equal size.. In other words there is no way to "tell" the function which datapoint comes from each speaker.. $\endgroup$
    – vggls
    Jan 26, 2022 at 15:25
  • 1
    $\begingroup$ @john_b I understand your motivation to try to not overfit on certain speakers. However, I wouldn't worry about it to start off with. K-fold cross validation randomly selects points to put in each fold, so presumably there will be a nice mix. Additionally, you could/should do repeated k-fold cross validation, so that you get the randomization multiple times. Finally, you can look at the CV performance vs. test set performance by specific speaker to see if there's any significant differences/overfitting by speakers. I suspect it won't be a big issue (unless diff. b/t speakers is big) $\endgroup$ Jan 26, 2022 at 15:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.