I am currently comparing RNNs to CNNs on multiple criteria and I've come across something I find rather weird.
I have a 3dCNN and an LSTM. Both take a 29x30x60 input and classify it to 2 classes. For the LSTM the input is flattened per timestep so it receives a 29x1800 input.
- The LSTM has +- 250k parameters.
- The 3dCNN has +- 100k parameters.
Besides architecture, everything is exactly the same (optimizer and learning rate etc.). However, the LSTM does this in around 100 epochs and the 3dCNN does this is around 5 epochs.
But this is where the weird part comes in: an epoch for the LSTM is about 30 seconds, where an epoch for the 3dCNN is around 45 MINUTES! The CNN goes to near 100% accuracy in about 10 epochs, where the LSTM does this in around 50-70 epochs.
Thus, the 3dCNN might train quicker epoch-wise, but is extremely slow in general compared to the LSTM. I did not expect this at all.
Yes I am training on a slow CPU in both cases but the difference in training time is my main concern here.
Is it the convolution that is so extremely more computationally expensive? Does anyone have a more detailed insight to this time difference in processing?
Edit: sorry for not including my network architecture.
My CNN is using 3 layers of convolution+maxpooling with 3D convolution, all using 3x3x3 kernels (stride 1 and zeropadding). The maxpooling is used on a 2x2x2 area. 1st layer of convolution extracts 16 feature maps, the 2nd 24, and the 3rd 32. This is followed by a dense layer of 16 nodes and a final dense layer of 2 nodes that perform the classification. The input is 29x30x60.
My LSTM uses 2 layers of lstm, with both layers having an output size of 16. The first is returning all its hidden states and the 2nd only returns its last hidden state. This is followed by a dense layer of 2 nodes. Input is 29x1800.
Hope this helps!