Feature engineering for sheet music

Question

I have a large dataset of digitized music scores that I'd like to use as input to a network. Initially, I'm looking to train networks to identify key signatures, tempo, dynamics, etc. from the raw streams of notes. This is to establish a baseline for future models for which I'd like the input to be raw audio files instead of nice, clean, discretized note inputs.

Given the temporal nature of music and the fact that scores are of different lengths, my initial high level sketches of such a network include LSTMs to handle sequences of "musical events" that occur sequentially in the scores.

I'm currently toying with exactly how to represent these musical events. I've arrived at the conclusion that different models (ex. tempo vs key signature detection) are better suited by different inputs.

Looking specifically at key detection, I think an ideal input is tempo and spelling invariant. Tone is most important, followed by rhythm. Ambiguities introduced by tempo or spelling feel like they just increase the amount of training data needed to make the model generalize. For example, the following when performed all sound identical:

Ideally, the input represents all of these equivalently. But also, changing the tempo of any of the above does not change the key, so the input representation should remain the same even in the face of tempo changes.

So my question is, how can I construct feature vectors with these properties (and why do such constructions have these properties)?

I substantiate this and demonstrate my thinking on this so far by walking through my research. It's rather long, but hopefully it demonstrates that I've put a lot of thought into this, but have hit a wall that requires some feature engineering expertise and insight that I don't have. Reading the below isn't particularly necessary if you know of some prior art or have a good approach for feature engineering music scores.

Research follows:

A naïve approach is to emulate MIDI input.

A quick summary for those unfamiliar with MIDI: MIDI encodes music as a sequence of events. Simplifying a tad, each event is a 3-tuple (onset_time, note, on_off). Each note has an "on" event and an "off" event. For example, a C4 lasting 0.25 seconds immediately followed by a G4 lasting 0.5 seconds would be encoded as two events: (0, C4, on), (0.25, C4, off), (0, G4, on), (0.5, G4, off) (note that the onset time is relative to the last event—not cumulative—and may be 0—indicating concurrent events).

A score would be a sequence of these event vectors. The tone would be a one-hot vector with the time and on/off (as a boolean) appended.

While simple, MIDI has representation ambiguity in the time domain. In particular, events with onset 0 can be in any order. Consider a C major triad: the notes C4, E4, G4 played at the same time. The on events for this could be any of these 6 orderings:

• (0, C4, on), (0, E4, on), (0, G4, on)
• (0, E4, on), (0, C4, on), (0, G4, on)
• (0, E4, on), (0, G4, on), (0, C4, on)
• etc.

I fear that a model would need to be presented with every possible ordering to generalize well. This could be overcome by pre-processing the input to, say, sort concurrent events by tone, so they were always in a predictable order. You could also coalesce concurrent events into the same event (make the note value vector not one-hot). However, even if you only feed good, pre-processed data into the model, it still feels that the input domain permitting such ambiguities could hamper learning efforts (is this suspicion unfounded?).

The on/off is of slight concern. I've observed before in simpler predictive models that getting an off note to be emitted after every on note was tricky. The MIDI format (and this representation) allows for you to turn notes on without ever turning them off (even though this essentially never should happen). I'm unsure if this lack of constraint on the input is detrimental.

A more salient concern is that onset times can be widely variable and, critically, depend on tempo and spelling (if we disregarded tempo, the first two of the above images would have different onset times because of note duration). Providing the model with times will require training on lots of tempo variations/respellings to ensure generalization.

I believe the onset issue rules out this MIDI-like approach.

One high level piece of information that is useful for key detection is knowing all the notes being played at a current time. Obviously, an LSTM is capable of remembering this, but this requires it to learn that it needs to remember this. We could help it along and remove some of the ambiguities by constructing an input that conveys this.

To understand this, you need to understand subdivision. Skip if you're familiar with this, I'd like to make this question approachable to those who don't read sheet or play.

---

Subdivision is the idea of dividing a rhythm into its component parts. Visually, it's the idea that the following when performed sound equivalent:

On top, there are two half notes. Below are 2 sets of 2 tied quarter notes. Quarter notes have a duration half that of the half note. The line connecting them is called a tie and it simply means to play them as if they were joined (don't stop playing in between them). Here we subdivided the half notes into quarters. It's kind of like saying 2 + 2 is equivalent to (1 + 1) + (1 + 1).

Musical notation follows this pattern of halving, so you can always subdivide notes. The notes from longest duration to shortest (where each note is half the duration of the previous) are: whole, half, 8th, 16th, 32th, 64th, etc. (practically, you can probably stop at 64th, but in theory it continues).

---

So we begin by normalizing the input to the longest note duration (ex. if the longest duration is a half note, we turn half notes into whole notes, quarters into halves, etc.). Then we subdivide to the shortest note length (ex. if the shortest duration is an 8th, then every quarter note is replaced with 2 8ths, every half is replaced with 4 8ths, etc.). Now we emit a 128x2 event vector for each subdivision. The first column is one of booleans indicating which of the 128 playable notes (technically, there are less than this, but MIDI allows for 128) are being played during that event. The second column is one of booleans indicating which of the notes were tied over from the last event. For example, if you subdivided to 8th notes, to represent a quarter note C4 you'd set the C4 row in the first column of the first event to true. In the second event, you'd set the C4 row of the first column and the second column (to indicate the tied note). Visually, a measure like this:

Would be subdivided like this:

And would be represented with events like this (we subdivide to 8ths, note that I also omitted the tones that aren't present and the events are separated by vertical lines):

$$\left [ \begin{array}{c|cc|cc|cc|cc|cc|cc|cc|cc} & \tiny{on} & \tiny{tie} & \tiny{on} & \tiny{tie} & \tiny{on} & \tiny{tie} & \tiny{on} & \tiny{tie} & \tiny{on} & \tiny{tie} & \tiny{on} & \tiny{tie} & \tiny{on} & \tiny{tie} & \tiny{on} & \tiny{tie} \\ \vdots \\ G4 & 0 & 0 & 0 & 0 & 0 & 0 & 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\ \vdots \\ F4 & 0 & 0 & 0 & 0 & 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\ E4 & 0 & 0 & 1 & 0 & 0 & 0 & 0 & 0 & 1 & 0 & 1 & 1 & 0 & 0 & 0 & 0 \\ \vdots \\ D4 & 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 1 & 0 \\ \vdots \\ C4 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 1 & 0 & 0 & 0 \\ \vdots \\ \end{array} \right ] $$

Notice how the quarter note E4 (second to last note) is two events (the 5th and 6th) with the latter having the tie bit set.

Now, this isn't the most compact representation (memory is perhaps a problem?). However, it lacks any of the tempo or phrasing related ambiguities. It does introduce some ambiguity in that the tie column is meaningless if it is set but its note from the adjacent column is not set (the note is not played during that event) or the note from the previous event's adjacent column is not set (the note was not played during the last event). However, this is countable ambiguity instead of the uncountable ambiguity introduced by the real onset parameter in the MIDI representation.

There is a fatal flaw with this representation, though. It cannot represent tuplets. Quick interlude for those not familiar:

---

A tuplet is a way of expressing note durations that aren't some power of 2 division of the beat (4 beats, 2 beats, 1 beat, 1/2 beat, 1/4 beat, etc.). In notation, this looks like:

The half note (last note) is a C5 lasting 2 beats. The three notes to the left of it (with the line with the "3" under them) are a triplet. Together the three of them also last 2 beats in duration. Each evenly divides these 2 beats, so each note in the triplet lasts 2/3 of a beat.

In theory, any number could replace the 3 (indicating the number of notes to divide the total duration—in this case, 2 beats—by). So, what non power of 2 fractions of a beat you have depend on the kind of tuplets in the score.

---

We could modify our subdivision process to instead divide by the least common multiple of all tuplet fractions and the smallest power of 2 subdivision. The problem here is even for triplets (and quintuplets, suptuplets, 11-tuplets definitely occur in reasonable music) the number of events it takes to represent a beat quickly gets out of hand. If our above example was instead written:

We could not longer subdivide to 8th notes. Instead, we'd have to subbdivide to 1/6 of a beat (the eighth notes are 1/2 beat subdivisions, but each note in the triplet is 2/3s of a beat— 1/6 evenly divides both). So now all eighths need 3 events. Before, our measure needed only 8 events. Now, it needs 24. I am concerned about this explosion for several reasons:

• It limits working with things in memory (considering most scores have 100 or more measures at 24 events apiece, each taking 128*2*1 byte = 256 bytes of memory means one score takes up 25KiB)
• For similar reasons as above, it severely limits batching as VRAM is more constrained then DRAM
• It introduces locality issues. While tempo and phrasing are no longer issues, LSTM memory must be longer depending on the amount of subdivision. In the first example, the LSTM needed to only remember the last 8 events to capture the whole measure. In the last example, the LSTM needs to remember 24 events to capture the previous measure. And this factor is dependent on the input (a piece with an 11-tuplet will need 1/22 of a beat subdivisions or 88 events per measure in 4/4—it quickly gets untenable).

A possible solution to this is to just round the triplet note length to a multiple of 2. So for example, above we could say that 2/3 is really close to an eighth note (1/2) + a sixteenth note (1/4 = .75) and just round it. I fear this information loss could be detrimental though. If the model, for example, developed an understanding of time signatures and measures, this rounding would break this as it would make measures with triplets 0.75 - 2/3s of a beat too long (and since key signatures change almost always only at the start of measures, inferring this information would be relevant to the model).

You could even make the case that 64th note subdivisions (requiring 64 events per measure in 4/4) is bad for the above reasons as well. Perhaps it would be better to subdivide all songs to the same fixed duration (say 32nds). However, this doesn't preclude the tuplet problem nor does it help with the wasteful use of memory.

Another wrench in the mix is grace notes. MIDI also struggles a little with this, as the actual duration of the grace notes is likely (MIDI generator) implementation dependent. I guess if we subdivided to something small enough (like a 64th), we could just add grace notes to previous events. Importantly, we wouldn't want to create new events with just them as this would affect the equal stepping of time between events. This doesn't work if the grace notes occur at the beginning of a piece and it certainly isn't ideal because it conflates grace notes with whatever subdivision we choose to represent them as (it's common for grace notes to be a chromatic run up to a note—this means often they are not in key whereas most other notes will be, which may be important information for the model).

For these reasons, I feel like this representation also isn't very good. Are there ways to improve either of these representations (given the constraints I mentioned above regarding ambiguity, memory, and LSTM memory length requirements)? Are there better representations for sheet music (I had trouble searching for prior art here, everything I found was general advice about feature engineering)?

---

As an aside: I recognize that the original MIDI proposal with one-hot vectors also aren't super space efficient. You have 128 bytes per note, which quickly adds up (although, I believe certain frameworks have efficient representations of one hots using sparse matrices—which makes this approach much smaller). But, say we subdivide to 64ths (ignoring tuplets for now). In 4/4, that's 32 events per measure (at 256 bytes per event). This means you'd have to average more than 64 notes per measure for the original MIDI proposal to occupy more space. This makes sense where you're merging voices from a large orchestra (where if there are say 16 instruments, each then can only average 4 notes per measure). But, for a solo piano piece, it doesn't. We must settle on one input, but the tradeoffs of these two both seem unacceptable.

Answer 1

Maybe useful in this context, also a little bit too late: A novel way to measure similarities between notes

Taken from here:

The use of the twelfth root of two in music is very well known in Western music. It is known in music theory that two successive pitches $a,b$ which sound “consonant” or “nice” if some ratio $B/A$ is “simple”. The notion of simplicity has not been defined precisely, and we will give a possible notion here:

Let $\alpha = 2^{\frac{1}{12}}$, $p_1 = \alpha^{k_1},p_2=\alpha^{k_2}$ where $0 \le k_1,k_2 \le 127$ are the midi pitches. We define the similarity between $p_1$ and $p_2$ to be: $$K_p(k_1,k_2) = \frac{\gcd(a,b)^2}{ab}$$ where $a = $ numerator of a rational approximation of $\alpha^{k_1-k_2}$ and $b = $ denominator of a rational approximation of $\alpha^{k_1-k_2}$. We argue that this similarity could capture when two pitches have a "simple" ratio and hence will sound "nice" together or when played in successive order.

Since a pitch alone does not describe a note, we have also defined similarity measures for duration, volume and if it is a rest or not:

Herefore we make use of the Jaccard-similarity of two intervals:

$$J(A,B) = \frac{\mu(A \cap B)}{\mu(A \cup B)} = \frac{\min(a,b)}{\max(a,b)}$$

where $A = [0,a],B = [0,b]$ are closed intervals and $a,b>0$ and $\mu([x,y]) = y-x$.

Using $J$ we define the duration similarity:

$$K_d(d_1,d_2) = J([0,d_1],[0,d_2]) = \frac{\min(d_1,d_2)}{\max(d_1,d_2)}$$

for two durations $d_1,d_2$ given as multiple of quarter notes. And similarily we define the volume similarity as :

$$K_v(v_1,v_2) = J([0,v_1],[0,v_2])=\frac{\min(v_1,v_2)}{\max(v_1,v_2)}$$

for $0 \le v_1,v_2 \le 127$ giving the volumes in midi notation. For rests we take the similarty $=0$ if one is not a rest and the other is, or $=1$ if both are no rests or both are rests.

Having two notes $n_1 = (p_1,d_1,v_1,r_1),n_2 = (p_2,d_2,v_2,r_2)$ we define a similarity between them as:

$$K(n_1,n_2) = \alpha_p K_P(p_1,p_2) +\alpha_d K_d(d_1,d_2) + \alpha_v K_v(v_1,v_2) + \alpha_r K_r(r_1,r_2)$$

where $\alpha_p+\alpha_d+\alpha_v+\alpha_r=1$ and $0<\alpha_x<1$ are weights, which can be chosen by the composer.

The mathematical properties of this similarity measure are also nice and can be proven. We can use this similarity measure to define a distance between two notes:

$$d(n_1,n_2) = \sqrt{2(1-K(n_1,n_2))}$$

This has the advantage of using the nearest neighbors algorithm in generating music. To capture similarities between fixed length sequences of notes, one could define the mean of the similarites:

$$K_S((n_1,\cdots,n_s),(N_1,\cdots,N_s)) = \frac{1}{s}\sum_{i=1}^s K(n_i,N_i)$$

This could be useful for measuring consonance of two melodies or so.

After having a kernel defined on notes, you could do Cholesky decomposition on the Gram matrix to get the feature vectors. Related question: A new method for processing music scores?