Generate music using LSTM using language model overfitting problem

Question

I want to try generate music using LSTMs from MIDI data. The model is based on the prediction of the next notes based on the previous sequence - based on known language models eg. char-rnn. To train I use 24 midi files of Chopin preludes.

1. I parse midi files into notes notation eg. b d e
2. Next I must create sequences and notes The output for each input sequence will be the first note.

For example for this plelude X will be sequences of notes and y it's predicted note after sequence.

          X                   Y
['C2', 'G3', 'G2', 'C4', 'E3'] => G4
['G3', 'G2', 'C4', 'E3', 'G4'] => E4
['G2', 'C4', 'E3', 'G4', 'E4'] => C4
['C4', 'E3', 'G4', 'E4', 'C4'] => A4
['E3', 'G4', 'E4', 'C4', 'A4'] => A3
['G4', 'E4', 'C4', 'A4', 'A3'] => B1
['E4', 'C4', 'A4', 'A3', 'B1'] => G3
['C4', 'A4', 'A3', 'B1', 'G3'] => G2
['A4', 'A3', 'B1', 'G3', 'G2'] => D4
['A3', 'B1', 'G3', 'G2', 'D4'] => F3

I put this data into neural network. I train model on NVIDIA Tesla K80 on 4h. The output melody is nice but it's not perfect. The problem appeared when I divided into a test and teaching set. The charts show that the model is overfit. I do not know how to improve it. I have already reduced the number of cells and added a larger dropout. The charts show that my model is overfitted, how can I fix it? It's a problem?

This is model architecture.

 model = Sequential()
    model.add(LSTM(
        256,
        input_shape=(network_input.shape[1], network_input.shape[2]),
        return_sequences=True
    ))
    model.add(Dropout(0.3))
    model.add(LSTM(512, return_sequences=True))
    model.add(Dropout(0.3))
    model.add(LSTM(512))
    model.add(Dense(256))
    model.add(Dropout(0.3))
    model.add(Dense(n_vocab))
    model.add(Activation('softmax'))
    model.summary()
    model.compile(loss='categorical_crossentropy', optimizer='rmsprop', metrics=['accuracy'])

    filepath = "weights-improvement-{epoch:02d}-{loss:.4f}-bigger.hdf5"
    checkpoint = ModelCheckpoint(
        filepath,
        monitor='loss',
        verbose=0,
        save_best_only=True,
        mode='min'
    )
    callbacks_list = [checkpoint]

    history = model.fit(network_input, network_output, validation_split=0.33,
                        epochs=600, batch_size=64, callbacks=callbacks_list)
    print(history.history.keys())

    # acc history
    plt.plot(history.history['acc'])
    plt.plot(history.history['val_acc'])
    plt.title('model accuracy')
    plt.ylabel('accuracy')
    plt.xlabel('epoch')
    plt.legend(['train', 'test'], loc='upper left')
    plt.savefig("acc_history.png")
    plt.close()

    plt.plot(history.history['loss'])
    plt.plot(history.history['val_loss'])
    plt.title('model loss')
    plt.ylabel('loss')
    plt.xlabel('epoch')
    plt.legend(['train', 'test'], loc='upper left')
    plt.savefig("history_loss.png")

How can I regularize this? It's possible?

Answer 1

With informations that you give, I would suppose 3 possibilities :

1. You don't have enough data to generalize, maybe use more data/other preludes depending to your problem

2. Maybe your training/test set don't have the same distribution, which means you learn a representation which not represent your testing data. Check distributions.

3. You overtrain your network, use Early stopping : with your data, the best configuration of your network is around 100 epochs.

First I would try to check distributions of my datasets, then increase dropout (to 0.5) and then try to get a larger dataset.

Answer 2

Since the objective of your project is to generate, You dont need to split your data to training and test, you can use all your data to better train your model.