# What features are suitable when predicting user preferences for songs?

I have a data set consisting of 1240 audio files (30 seconds each) and a file like this (two first rows):

u  v   decision
1  323 0
12 9   1


u and v are different audio IDs, whereas decision is which one of the songs the user preferred.

I want to build a GLM that can predict the user's decision, based on the input of two songs. However, I am not sure what features to extract from the songs, as I don't have much experience with audio files. As of now, I have converted the audio files to gammatone-based spectrograms.

My design matrix consists of two gammatone-based spectrograms subtracted from one another (the two songs I am comparing).

Does it make sense? Does anyone know specific features that are relevant for such problems?

• Interesting problem. I'm curious for your final solution. – Daniel Dostal May 6 '18 at 11:15
• Genre, artist, how much played on the radio/YouTube, tempo, key, singing (y/n or percentage of time) all seem like reasonable ideas. – Björn May 6 '18 at 13:38
• If this is the data that you have (audio/spectrogram only) then the above information doesn't seem known/relevant. – PixelatedBrian May 6 '18 at 22:47
• For a spectrogram being fed into a linear model the image would basically be a large 2d matrix (could flatten to 1d, the model shouldn't care in this case) with each pixel being a feature. To get something that a linear model can run faster OP might want to consider trying PCA to reduce the dimensionality of the features being fed into the GLM. example of concept – PixelatedBrian May 6 '18 at 22:55
• @PixelatedBrian do you think it makes sense to subtract the reduced spectrograms (from applying PCA) from one another? – lala_12 May 7 '18 at 11:21

Try MFCCs (Mel-Frequency-Cepstrum-Coefficients). They are widely used and you can find a lot of examples to get a good understanding and references about useful parameterisation: See i.e. the project on spotify content based recommendation http://benanne.github.io/2014/08/05/spotify-cnns.html).

Gammatone filters are probably suitable too, but they are a lot more complex / harder to explain / supposedly computationally expensive

For further information on audio feature extraction see the online course of Alexander Lerch: https://www.audiocontentanalysis.org/teaching/

Why do you want to subtract the spectrograms? More precisely, why should the amount the spectrograms differ from each other control the outcome of the choice?

Lets substitute music for cake in this experiment. Our participant will be presented with two options of cake. As with the music we have 1240 different recipes. We know the ingredients for each, and we also have some data about past choices of our participant.

If we are lucky, we find i.e. that the participant always took the cake with less sugar, and has high preference for those with a lot of vanilla. With that information we can give ratings to all our recipes (low rating for high sugar, high rating for high vanilla). Our prediction for the participants choice now is a mere comparison of which of the two recipes has the higher rating.

If we however subtracted one recipe from the other, this implies something like: "if the cakes largely differ in amount of sugar, participant will take a, otherwise participant will take b". I am not saying that this is not a possible effect, but from my every day experience it will definitly not be the strongest.

In one sentence: Do not subtract the spectrograms, but use them before hands to create song ratings that you can then compare between two given examples.

Instead of predicting which features are relevant up front, extract a wide range of features, then use your data to filter out the relevant ones. Since your labels are on the whole song, start with features which are (summarized) per-song.

Essentia implements a huge range of feature extraction algorithms for audio, including many that are specific to music.

Besides doing exploratory data analysis on the features, a RandomForest classifier can be useful starting point, since it can indicate feature importance and handles correlated and noisy features well.