I'm working on a project to classify 30 second samples of audio from 5 different genres (rock, electronic, rap, country, jazz). My dataset consists of 600 songs, where the features are a 1D array of mfccs for each song and the labels are the genres. The first 1/5th of the feature array is rock, the 2nd 1/5th is electronic, and so on.

I am focusing on first trying out my dataset on an svm classifier. To find the best set of parameters, I am using scikitlearn's gridsearchcv function, with the number of folds set to 5.

Before doing a gridsearch, I am splitting up the 600 songs randomly into train and test sets, with the test set being 20% of the 600. I randomly select 20% from the entire 600 as test and 80% as train. My question is, is this an appropriate method? Or should I be selecting a random sample from each genre, so as to make sure that my training set always contains the same amount from each label?


Either way is fine. When the label set is unbalanced and especially when the total dataset size is relatively small, it might be useful to split equally among different classes – this is called "stratified" sampling, and scikit-learn has methods for this. But 5 equal classes and 600 total samples means that you'll probably get a reasonable number of each in your split, so you don't really have to worry about it.

| cite | improve this answer | |
  • $\begingroup$ Ok cool. What is the more typical approach, usually? $\endgroup$ – ohbrobig May 15 '17 at 0:07
  • $\begingroup$ Probably the default is not to stratify unless you have a small or very unbalanced sample, but I don't know if there's any strong reason for that. $\endgroup$ – djs May 15 '17 at 0:08
  • $\begingroup$ Given that my dataset isn't that big, would this be a wise move? While almost always the rbf kernel is better than the linear, I'm noticing on repeated runs sometimes linear SVMs perform almost as well as an rbf. I'm thinking this might be due to my training set not always being the best representation. $\endgroup$ – ohbrobig May 15 '17 at 0:20
  • $\begingroup$ Additionally, given my somewhat small dataset, before doing 5 fold cv what do you suggest: a) calling train_test_split wtih random sampling and stratifying or b) calling the function without random sampling and stratifying $\endgroup$ – ohbrobig May 15 '17 at 1:01
  • $\begingroup$ RBFs aren't necessarily always going to be better than linear, especially for a relatively small dataset. I'd probably stratify if you're worried about it, but I don't expect to see a substantial difference either way. $\endgroup$ – djs May 15 '17 at 11:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.