Following this example in matlab, I'm trying to apply the same principle on my problem. Disclaimer: I am a total beginner in this field, I'm starting to study and learn it, so be kind, patient and simple in explaination!
I have 14 sequences of features, of variable length, representing a physical object. There can be 4 ways to look at this object, therefore I need to classify four categories / known similar sequences. These sequences are sampled in space, and I have a 15th column with the space position, if needed. Those matrices of data span from 14x100 to 14x300, more or less. They can be rich in zero values (even all zeros in one or more features), and they are in the [0, 1] interval. This is an example picture of one sequence: I can accepth both passing a sequence slice at time, having a feedback if it's a recognized sequence (preferred), or buffering and passing sequence blocks with a sliding window.
I have roughly 300 samples [74 80 79 69] of data well labeled, and I split it into 85% train and 15% test; unfortunately right now matlab does not support validation and lstm. I implemented the network as the example, adjusting it to my case (right now I work with full sequences). Here it is my code:
layers = [ ... sequenceInputLayer(14) %my inputs bilstmLayer(opts.numHiddenUnits,'OutputMode','last') fullyConnectedLayer(4) %the outputs softmaxLayer classificationLayer]; options = trainingOptions('adam', ... 'ExecutionEnvironment','cpu', ... 'GradientThreshold',1, ... 'MaxEpochs', 200, ... 'MiniBatchSize', 20, ... 'SequenceLength','longest', ... 'Shuffle','never', ... 'Verbose',0, ... 'InitialLearnRate', 0.001, ... 'LearnRateSchedule', 'piecewise', ... 'LearnRateDropPeriod', 70, ... 'LearnRateDropFactor', 0.1, ... 'Plots','training-progress');
As suggested in the example, I sort the sequences by length to reduce padding, and when I prepare the dataset I shuffle the sequences to extract different train sets, in order to verify that the net is well generalized everytime I run a training. I have several problems that I don't understand, therefore I don't know how to tackle them.
- Performing the same training multiple times yelds a very different test accuracy, I went from 0.6 to 0.95. I tried changing the batch size, from 1 to the full dataset, or the number of hidden units, but this variance still shows up. Considering the random selection of the training set, and the weights random initializing, can this be because I have too few or too similar training examples? I noticed that all the categories present this variance, with the 2nd being especially troublesome.
- The accuracy keeps oscillating quite a lot during training, with vastly different results from train to train (see pics). I tried adjusting the learning rate and the drop factor, or the batch size, but still I cannot obtain something more stable.
- The accuracy during the training differs quite a lot from what I get in the test phase. I guess the net is overfitting, but I cannot use the validation set because matlab 2018a does not support it. Is my guessing correct? What steps can I take to mitigate this problem? I know about regolarization, but I don't know how to do it in practice.
I guess that the training took long enough, because the loss seems stable to me, but I don't have a grasp on how all the other parameters must be set up, and what are the effects of those. Plus, I really have no clue about how many layers and how many units per layer must be used. Can you all point me some clear tutorials about that?
If I was not clear, please tell me where I need to clarify my question. Many thanks!