How to reduce dimensionality of audio data that comes in form of matrices and vectors?

Question

I'm working on a project involved with identifying different types of sounds (such as screams, singing, and bangs) from each other. We've got our data a reasonable number of different transformations (e.g.: spectrograms, chromagrams, MFCCs, etc.), but since most of our features are 2-dimensional matrices (some are actually 1-dimensional vectors), we'd like to reduce this information in some way, so that the machine learning we're hoping to do takes a "feasible" amount of time. However, I don't quite know enough about math and statistics to make an educated decision on this.

Our data consists of small sound files from ~1-10 seconds long. There are recordings of screams, singing, bangs (and other man-made noises), and birds (and other natural noises). We are hoping to be able to differentiate and identify each source type from the others. See https://github.com/BenSandeen/surveillance_sound_classifier/blob/master/Project.ipynb for the different plots we have made to guide our selection of features to use. Focus mainly on the 3x3 plots, as that's where the comparisons are being made. These plots are primarily time vs. frequency, with amplitude represented by color.

I was thinking that maybe we could "collapse" each matrix down to a vector by somehow choosing some representative frequency/amplitude-related feature at each time slice (we're using Short Time Fourier Transforms to analyze the sounds) and then get a vector of some length, containing a bunch of scalars. Although this could make accounting for different lengths of sounds difficult. Would it be reasonable to just set the shorter sounds' vectors to be filled in with zeros if they have no useful data? That would effectively make these sounds a projection onto some lower-dimensional space. Then, maybe we could just use dot products to compare the vectors; if they're parallel, they'll have a large for product, but of they're nearly perpendicular, they'll have a dot product near zero.

Alternatively, I was thinking that something like a trace of our matrices, or finding their characteristic polynomial, might be useful direction to pursue. I've read a bit about PCA, but I'm not quite understanding it enough to know if this may be what I'm looking for.

Can anyone think of any other ways of handling and reducing this data? For what it's worth, we're currently planning on using Sci-Kit Learn (sorry that I can't use more than 2 links yet) to perform our machine learning.

Answer 1

Looking at the task described I think you are dealing with a somewhat classical problem of classifying images albeit in more abstract form. In your case you want to classify spectrograms rather than standard images. I would suggest you look at image classification or spectrogram classification tutorials online.

Having said this general advice: A standard way to reduce your data would be to use principal component analysis. I think you tried to do somewhat gave up half-way in the notebook you attach. In any case, you would do something like this:

1. Transform your matrix dataset into vectors. (ie. if you have 100 50$ \times$ 50 matrices, you will now have 100 2500-element vectors; see here for more information on the equivalence of two-dimensional PCA to line-based PCA)
2. Use PCA to derive the principal components $\phi$ (You will possibly deal with a $n << p$ situation, see here and here for some first steps).
3. Use $\phi$ to get the principal component scores $\xi$ (the projected data essentially).
4. Use $\xi$ as surrogate data for your original sample, ie. you classify your sample using these. (I would personally suggesting using a logistic regression first before moving to more exotic beasts (eg. decision trees, boosting, etc.))

The only real restriction you must take care of is the need for the matrices holding your spectrograms to be of the same sizes. You can achieve that in multiple ways; you need to either define a representative utterance time (so you practically interpolate all your sample onto a common time-grid) or you segment your utterance accordingly. Both options are slightly messy because you will need to do quite a bit of data wrangling. I have worked with human utterances in the past and that was somewhat "easy" because there is the concept of a word (usually) but the principal is the same; you want the space over your data are recorded to be the same for all data and if possible to have a physical interpretation.

You may want to consider using 2D Wavelets as a basis instead of the principal components suggested by PCA. 2D wavelets are localised on time and frequency so that should give you a more informative representation than say a Fourier polynomial basis that is focusing on frequency only. I am not a frequent Python user but Python seems to have some implementation on this matter already (eg. pywavelets) if you want to experiment on this.

Additional points:

1. Check the noise in your sample. Usually statistical samples are assumed to have the same amount of "noise" in them (by noise I mean unstructured variance). Recording almost always do not have this luxury. Different recording environments, different equipment, etc. can lead to a lot heteroskedasticity. You will probably need to denoise your data first.
2. Check your sample for frequency artefacts. Sound recording are finite representations of a infinite process (a wave). A large diverse sample usually has recording on different sampling frequencies. Before starting you want all your sampling frequencies to be similar because otherwise you risk aliasing artefacts.