I have a dataset of energy measurements taken every minute from the energy footprint of home appliances. Based on that I am trying to detect human presence in the house. Since the data is sequential, I decided to use a recurrent neural network (LSTM) to model it.
Because the classes are unbalanced (~60/40) I am using confusion matrix metrics along with accuracy. The other main characteristic of the dataset is that it has a great deal of sparsity, especially in certain features.
I have a baseline of a deep feedforward network to compare the model against. During training of the feedforward variant I shuffle the dataset to return it to an iid setting and prevent overfitting.
I am using the negative log likelihood as my cost function in both cases. This is a binary classification setting so this, in turn, yields the binary cross entropy cost.
The feedforward network yields the following results:
accuracy: 86.37%
f1 score: 82.00%
precision: 82.98%
recall: 81.10%
The recurrent network, on the other hand, yields the following:
accuracy: 31.08%
f1 score: 38.35%
precision: 68.53%
recall: 58.34%
Clearly, the LSTM is performing significantly worse. My intuition is that it is overfitting the data (the training cost is decreasing steadily). I should mention that I do not shuffle the dataset during training of the LSTM, as I want to preserve the sequential structure of the data. Am I making wrong assumptions about training it? Should I actually shuffle the dataset? If that is not the problem, does anyone have any pointers on what could be going wrong?
I also thought sparsity is a problem but then again it doesn't seem to affect the feedforward variant too much.
Thanks in advance.
