3
$\begingroup$

I have a dataset of energy measurements taken every minute from the energy footprint of home appliances. Based on that I am trying to detect human presence in the house. Since the data is sequential, I decided to use a recurrent neural network (LSTM) to model it.

Because the classes are unbalanced (~60/40) I am using confusion matrix metrics along with accuracy. The other main characteristic of the dataset is that it has a great deal of sparsity, especially in certain features.

I have a baseline of a deep feedforward network to compare the model against. During training of the feedforward variant I shuffle the dataset to return it to an iid setting and prevent overfitting.

I am using the negative log likelihood as my cost function in both cases. This is a binary classification setting so this, in turn, yields the binary cross entropy cost.

The feedforward network yields the following results:
accuracy: 86.37%
f1 score: 82.00%
precision: 82.98%
recall: 81.10%

The recurrent network, on the other hand, yields the following:
accuracy: 31.08%
f1 score: 38.35%
precision: 68.53%
recall: 58.34%

Clearly, the LSTM is performing significantly worse. My intuition is that it is overfitting the data (the training cost is decreasing steadily). I should mention that I do not shuffle the dataset during training of the LSTM, as I want to preserve the sequential structure of the data. Am I making wrong assumptions about training it? Should I actually shuffle the dataset? If that is not the problem, does anyone have any pointers on what could be going wrong?

I also thought sparsity is a problem but then again it doesn't seem to affect the feedforward variant too much.

Thanks in advance.

$\endgroup$
1
  • $\begingroup$ Can you show how it works on training vs. validation? This should tell if overfitting is the problem. What can you tell us about number of rows (aka samples) and columns (aka different things measured) for your data. Is it just one day, or is it several? Is the training labelled, or are you trying to determine whether it is occupied? $\endgroup$ – EngrStudent Jul 3 '18 at 11:28