# How to prepare sequence data for input/output for a LSTM generative model?

I've a set of data containing midi message information such as pitch, velocity, note on/off and delta time between the messages. These messages basically just act like the different musical notes.

I intend to treat the list of messages (which essentially are the notes being played) as sequences and run it through a LSTM network to create a generative model. Something like what a character RNN might do. However, as opposed to a character RNN which has only 1 character of input/output, I'm have a long message vector to pack all of that pitch, velocity, etc in my input/output.

Sadly, my model isn't generating what I would expect with this design. I suspect this is because of the way I preprocessed the data that is adding complexity to the inputs and outputs.

Here's what I'm doing to each of the message in the midi file:

• These are the information in each message: pitch, velocity, note on/off and delta time
• Convert pitch into note and octave
• There are 12 possible notes and 10 possible octave
• So I turn the note and octave into 2 separate one-hot encoding vectors
• Then there is velocity. Velocity can range from 0 to 127.
• I turn the velocity into a 128 long one-hot encoding vector
• Next, there is the delta time which is the delay in time from the last note to the current note
• Because the time can be any integer value, such as 64, 850, etc, I left it as it is.
• Finally, the note on/off. A note on represents that a note will be played until a note off is message is triggered.
• It can either be 0 or 1 only. So if the value was note_on, I make it as 1, 0 otherwise.
• Then, I combine every vectors and values into one long message vector in this format: [12-long Notes OHE, 10-long Octave OHE, 128-long Velocity OHE, Time, Note On/Off]

As an example, a combined message vector may look like this:

[  0   1   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   1
0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0
0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0
0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0
0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0
1   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0
0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0
0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0
0   0   0   0   0 896   1]


So the combined message vector is 152 long and it actually contains these sections which I've highlighted them in different colours:

Yellow: Note 12-long OHE vector. So in the example, that was C#. Pink: Octave 10-long OHE vector Purple: Velocity 128-long OHE vector Dark Green: delta time Light Green: 0 for note off, 1 for note on

This will be one of the messages in a sequence. The sequence will have many other message vectors in this same format.

I then run this through a LSTM network in Keras to generate the next message vector using a timestep of 10. I'm also using an output of Dense(152) and relu as activation function. The model uses mean squared error as loss function and adam as optimiser function.

input = Input(shape=(1000, 152))
x = LSTM(128)(input)
x = Dropout(0.2)(x)
output = Dense(152, activation='relu')(x)

model = Model(inputs = input, outputs=output )


However, no matter how many epochs I run it, the model is always generating garbage vectors which don't even follow the format. It returns something like this:

[[  0.        ,   0.30502695,   0.        ,   0.        ,
0.        ,   0.        ,   0.20570225,   0.        ,
0.        ,   0.        ,   0.        ,   0.        ,
0.        ,   0.        ,   0.        ,   0.        ,
0.28584039,   0.37196404,   0.        ,   0.        ,
0.        ,   0.        ,   0.91128671,   0.        ,
0.        ,   0.        ,   0.        ,   0.        ,
0.        ,   0.        ,   0.        ,   0.        ,
0.        ,   0.        ,   0.        ,   0.        ,
0.        ,   0.        ,   0.        ,   0.        ,
0.        ,   0.        ,   0.        ,   0.        ,
0.        ,   0.        ,   0.        ,   0.        ,
0.        ,   0.        ,   0.        ,   0.        ,
0.        ,   0.        ,   0.        ,   0.        ,
0.        ,   0.        ,   0.        ,   0.        ,
0.        ,   0.        ,   0.        ,   0.        ,
0.        ,   0.        ,   0.        ,   0.        ,
0.        ,   0.        ,   0.        ,   0.        ,
0.        ,   0.        ,   0.        ,   0.        ,
0.        ,   0.        ,   0.        ,   0.        ,
0.        ,   0.        ,   0.        ,   0.        ,
0.        ,   0.        ,   0.        ,   0.        ,
0.        ,   0.        ,   0.        ,   0.        ,
0.        ,   0.        ,   0.        ,   0.        ,
0.        ,   0.        ,   0.        ,   0.        ,
0.        ,   0.        ,   0.        ,   0.        ,
0.        ,   0.        ,   0.        ,   0.        ,
0.        ,   0.        ,   0.        ,   0.        ,
0.        ,   0.        ,   0.        ,   0.        ,
0.        ,   0.        ,   0.        ,   0.        ,
0.        ,   0.        ,   0.        ,   0.        ,
0.        ,   0.        ,   0.        ,   0.        ,
0.        ,   0.        ,   0.        ,   0.        ,
0.        ,   0.        ,   0.        ,   0.        ,
0.        ,   0.        ,   0.        ,   0.        ,
0.        ,   0.        ,   0.        ,   0.        ,
0.        ,   0.        ,   0.        ,   0.        ,
0.        ,  25.11255646,   0.348]


This is not even close to what I would expect the output to be. The format is all wrong.

Is my problem with the way I arranged my data? But I'd to do this to pack all of that message information into the sequence. How should I arrange my data for this to work? Otherwise, what is the problem I'm having?

• You most likely need to incorporate masking to ignore the placeholder 0s: stackoverflow.com/questions/42353056/… – Alex R. Nov 12 '17 at 18:59
• @AlexR. Thanks, but those zeros in my case are not placeholders? Those zeros are from the one-hot encodings. I've just masking but I'm still getting that same kind of results. – Carven Nov 13 '17 at 0:06
• Try normalizing the delta time to be in range 0-1. Using a value of even just 100 will wash out all the other inputs, especially one as high as 896. Also try not using relu (or any activation function at all) on the output. It's never usually a good idea. Instead take your output and treat its max value as a one and set all the other values to zero for each individual input's area that has a one hot encoding. And lastly you could try not using one hot encoding at all and normalize all your inputs – Frobot Nov 15 '17 at 23:01