Generate music using LSTM using language model overfitting problem

Question

I want to try generate music using LSTMs from MIDI data. The model is based on the prediction of the next notes based on the previous sequence - based on known language models eg. char-rnn. To train I use 24 midi files of Chopin preludes.

1. I parse midi files into notes notation eg. b d e
2. Next I must create sequences and notes The output for each input sequence will be the first note.

For example for this plelude X will be sequences of notes and y it's predicted note after sequence.

          X                   Y
['C2', 'G3', 'G2', 'C4', 'E3'] => G4
['G3', 'G2', 'C4', 'E3', 'G4'] => E4
['G2', 'C4', 'E3', 'G4', 'E4'] => C4
['C4', 'E3', 'G4', 'E4', 'C4'] => A4
['E3', 'G4', 'E4', 'C4', 'A4'] => A3
['G4', 'E4', 'C4', 'A4', 'A3'] => B1
['E4', 'C4', 'A4', 'A3', 'B1'] => G3
['C4', 'A4', 'A3', 'B1', 'G3'] => G2
['A4', 'A3', 'B1', 'G3', 'G2'] => D4
['A3', 'B1', 'G3', 'G2', 'D4'] => F3

I put this data into neural network. I train model on NVIDIA Tesla K80 on 4h. The output melody is nice but it's not perfect. The problem appeared when I divided into a test and teaching set. The charts show that the model is overfit. I do not know how to improve it. I have already reduced the number of cells and added a larger dropout. The charts show that my model is overfitted, how can I fix it? It's a problem?

This is model architecture.

 model = Sequential()
    model.add(LSTM(
        256,
        input_shape=(network_input.shape[1], network_input.shape[2]),
        return_sequences=True
    ))
    model.add(Dropout(0.3))
    model.add(LSTM(512, return_sequences=True))
    model.add(Dropout(0.3))
    model.add(LSTM(512))
    model.add(Dense(256))
    model.add(Dropout(0.3))
    model.add(Dense(n_vocab))
    model.add(Activation('softmax'))
    model.summary()
    model.compile(loss='categorical_crossentropy', optimizer='rmsprop', metrics=['accuracy'])

    filepath = "weights-improvement-{epoch:02d}-{loss:.4f}-bigger.hdf5"
    checkpoint = ModelCheckpoint(
        filepath,
        monitor='loss',
        verbose=0,
        save_best_only=True,
        mode='min'
    )
    callbacks_list = [checkpoint]

    history = model.fit(network_input, network_output, validation_split=0.33,
                        epochs=600, batch_size=64, callbacks=callbacks_list)
    print(history.history.keys())

    # acc history
    plt.plot(history.history['acc'])
    plt.plot(history.history['val_acc'])
    plt.title('model accuracy')
    plt.ylabel('accuracy')
    plt.xlabel('epoch')
    plt.legend(['train', 'test'], loc='upper left')
    plt.savefig("acc_history.png")
    plt.close()

    plt.plot(history.history['loss'])
    plt.plot(history.history['val_loss'])
    plt.title('model loss')
    plt.ylabel('loss')
    plt.xlabel('epoch')
    plt.legend(['train', 'test'], loc='upper left')
    plt.savefig("history_loss.png")

How can I regularize this? It's possible?

Answer 1

With informations that you give, I would suppose 3 possibilities :

1. You don't have enough data to generalize, maybe use more data/other preludes depending to your problem

2. Maybe your training/test set don't have the same distribution, which means you learn a representation which not represent your testing data. Check distributions.

3. You overtrain your network, use Early stopping : with your data, the best configuration of your network is around 100 epochs.

First I would try to check distributions of my datasets, then increase dropout (to 0.5) and then try to get a larger dataset.