I'd like to create a model capable of emulating music that has been presented to it. The model ought to be specifically designed for that purpose, not just another generic, stacked LSTM. For the purposes of this question we'll only address generating chords, although I'll probably extend the model later if this is even remotely succesful.


Music, unlike many other sequences like text, is about relationships, not so much about absolute values. A problem with basic architectures is that relationships between inputs are not generalised well. So given sequences of (one-hot-encoded) inputs like 1-1-3 and 2-2-4 the model has no clue what comes after 3-3.

These two videos discuss different types of neural networks for music generation. [1, 2]

Initial idea

I do have an idea of what could be a good representation of a chord and an architecture to support that idea. Chords are agnostic about the pitch or voicing (order of notes) they are played in, and a chord cannot contain two notes that are the same. As information about those things is redundant at best, misleading at worst, we could dispense with that information.

There are a total of twelve notes in an octave, after which the notes repeat. Imagine a ring of those twelve notes. That ring represents every possible note collapsed into one octave (note pitch mod 12). Chords are placed onto that ring.

These qualities would be desirable for a model.

  • Learns from sequences, so predictions are heavily influenced by previous steps
  • Generalises on intervals (distances between notes), so sequences like 3-4-5, 7-8-9 and even 12-1-2 would be viewed as similar.

In the videos linked above, two architectures had a convolutional element to them. One was fully convolutional (HyperGAN), the other used convolution slightly differently (PixelCNN). I'd imagine some similar construct would be needed here.

Let me write my thoughts down: I'm not sure if a convolutional structure is needed or just something that resembles it a lot. In my mind I'm imagining a node for each interval, such that one node (or layer or heck even a network) would recognise shifts of 1, another shifts of 2, 3 and so on.


As my knowledge and intuition of convolutional and recurrent neural networks is limited to say the least, I'm not sure what would be the best structure to use here. Have you encountered this kind of need before? In music or elsewhere. I'd love any pointers or insight.

Is the structure described above suitable for modelling the problem? Would using straight-out convolutions change the situation, for better or for worse? If convolutions are used, can the gap in the ring of notes between 12 and 1 be bridged?

Why Cross Validated?

This is a design-heavy question, so it could go on Software Engineering, but the nature of the problem - machine learning - convinced me this would be a more appropriate place. The question addesses the high-level design, not the implementation on software level.

  • $\begingroup$ Maybe I am misunderstanding the question, but could you not represent a chord by a letter, instead of a combination of individual notes? Then your problem transforms into a sequence of letters which should be easy to model. $\endgroup$
    – GR4
    Commented Aug 28, 2018 at 10:11
  • $\begingroup$ @GR4 Yes, that's what I have going on now. But it does not capture the time series quite right. If I have the chord A minor with a major sixth (relatively common) or really anything besides a major chord, additional letters need to be introduced, whatever the notation. These additional letters would mess up the chord-per-timestep nature. If you mean encoding chords to letters regardless of their meaning, that doesn't quite capture the relationships between the chords. an A minor is very similar to C major, but as one-hot encoded inputs they are completely different. Did I answer your question? $\endgroup$
    – Felix
    Commented Aug 28, 2018 at 10:18
  • $\begingroup$ If I understand correctly, you mapped the chords to numbers in order to reflect their relationship in terms of pitch, as in A=1, B=2, etc. And your problem is that any chords that deviate from such relationship cannot be modeled with such a system, is that correct? $\endgroup$
    – GR4
    Commented Aug 28, 2018 at 10:26
  • $\begingroup$ @GR4 Not quite, what I have currently is a scheme where I write chords down (almost) as if notating them to be played. Then feeding the letters one by one to the model. But what I'm proposing here is to parse every chord down to its individual notes. From those notes we would construct a "many"-hot-encoding and feed that to the model. E.g. the chord C major would be represented in the chromatic scale starting from c as an array of on or off notes: 1-0-0-0-1-0-0-1-0-0-0-0. My problem is getting the model to recognise relationships between inputs as well as sequences within one input. $\endgroup$
    – Felix
    Commented Aug 28, 2018 at 10:35

1 Answer 1


I think you have two options:

  • Keep a representation of chords that is able to reproduce the complexities that you desire, for example several chords at the same time, such as the *kern representation or midi. These representations have the advantage that some complexities can be included such as multiple instruments, note durations and (for midi) even note intensities. The problem can then again be formulated as a time-series. A *kern file is in fact simply a text file that can be transformed into a sequence of symbols.
  • Transform all your notes to an array of on or off notes, as you suggest. There is no reason why you can't model this as a time-series in the sense that each timestep would simply have an entire embedding as an input that could be fed to your model (eg 1D convolutional layer or LSTM). The problem here is that, depending on the complexity of the music you would like to represent, you would probably need to implement the overhead yourself that adds the relevant information about note duration/instrument/etc and append it to your embedding as well.
  • $\begingroup$ Thank you for the answer, and sorry for the late reply! The first approach seems nice, but I think for just modelling chords it would not be optimal. I consider two chords played at the same time just a complex chord. And note intensities and individual durations are not really the target here either. But about the second approach: you'd suggest perhaps a 1D convolution. How could one bridge the ends together as described in the question? Jumping from the last note back to the first on the scale should not appear any different to other shifts of one. Cheers! $\endgroup$
    – Felix
    Commented Sep 1, 2018 at 7:36
  • $\begingroup$ I'm not sure why you think those ends wouldn't be bridged together with this setup. Given a large enough dataset, the model will learn that such shifts of one have a similar "meaning". I did some research in the past using the *kern representation and was able to show that one can train embeddings that have automatically learned the harmonic information of chords (eg the notes for the chord A are mapped in a similar configuration as the notes of the chord B) as well as other info such as note length & pitch. $\endgroup$
    – GR4
    Commented Sep 3, 2018 at 9:06
  • $\begingroup$ In summary, if you have a large dataset with meaningful objective, the network should be able to extract relevant features by itself. $\endgroup$
    – GR4
    Commented Sep 3, 2018 at 9:06
  • $\begingroup$ I've understood convolutions (with the minimal experience I have) to pool local features gradually up towards the scale of the whole data (image - or array in this case). My intuition says the opposite ends of that array wouldn't get pooled in the same manner, but maybe I'll have to do some research. Regardless, thank you very much for your answers! $\endgroup$
    – Felix
    Commented Sep 3, 2018 at 13:40
  • $\begingroup$ No, the 1d convolution only pools the local features in the time dimension. The embedding that you feed to the convolution is taken entirely into account, ie all notes. Also note that some CNN architectures have been shown to perform well on sequences taken info account over larger time steps, similar to LSTMs. $\endgroup$
    – GR4
    Commented Sep 3, 2018 at 22:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.