Training a Neural Network on Music

Question

I'm very picky about the music I like. I can't say I like a genre in particular because my tastes in music boil down to how a specific song sounds. For a while now, I've had a vision of creating a neural network that can tell me what songs I "would" like based on songs that I tell it I like.

Forgive me if any of my presuppositions or inferences about how neural networks is wrong. I have no experience with neural networks and I only somewhat understand how they work.

Let's say that I have a list of about 5000 songs. Some of these I really like, some I really don't like. I can "flag" the songs that I really like. Then, I can subject these songs to a neural network to train it. Several numerical properties could be extracted from the music, including peak amplitude, average amplitude, chordal analysis, harmonic data, and so on. The neural network "knows" which songs out of this initial data set that I really like, so I want it to find differences between those songs and the songs that I haven't flagged as "liked". Then, once trained, I envision feeding this neural network a new set of songs, and it predicts which songs out of this new set I would like based on what it learned from the initial data set.

Would this sort of thing be possible? If so, does anyone have any resources that might help?

Answer 1

Particularly for your question, a way of doing that is converting audios to spectrograms using FFT, then train a convolutional neural network. See a somehow old paper. And if you search spectrogram on GitHub, you can find tons of its applications in audio related problems like human speech recognition. For getting decent predictions, you'll need more sophisticated methods and larger datasets.

More generally, recommender systems are now indispensable for popular apps, like YouTube, Spotify, Tweeter, TikTok, Amazon and many more. I personally find that YouTube can fit my taste of music and videos quite well. You can find massive resources about it, from basics to bleeding-edge researches.

Answer 2

There is a ton of work out there involving machine learning approaches to music analysis. Here are a few suggestions:

• Music and Audio Research Laboratory https://steinhardt.nyu.edu/marl

• The Million Song Dataset http://millionsongdataset.com/

• The Music Matrix https://musicmachinery.com/2011/11/27/the-music-matrix-exploring-tags-in-the-million-song-dataset/

• plamere /msd-matrix-explorer https://github.com/plamere/msd-matrix-explorer

• tbertinmahieux/MSongsDB · GitHub https://github.com/tbertinmahieux/MSongsDB

• Frequently Asked Questions | Million Song Dataset http://millionsongdataset.com/faq/

Answer 3

You don't need a classifier for what you're doing in the first step necessarily. DNN classifiers consist of two parts, a feature extractor and a classification head. The feature extractor typically makes most of the networks parameters and the classifier head is often just a simple linear model. Basically you transform your data into a space where you can easily make decisions with a linear model. However, training this model jointly requires tons of labeled data. Instead, you could also use a feature extractor in an unsupervised fashion where you only require large amounts of unlabeled data. For images, people use methods like BEiT (BEiT: BERT Pre-Training of Image Transformers). I don't work in music but maybe this paper has some useful ideas. With such a feature extractor you could then train a classifier on songs you labeled or maybe even use some sort of recommender system (although those typically learn from larger databases of what songs different users like).