My CNN-LSTM EEG Keras classification model includes a Dense 'shortcut' connection for residual sequence learning as shown below; to match dimensionality, the Dense layer's set to
2*lstm_dim. When feeding very long sequences (>200,000 timesteps), this Dense layer dominates model size (# of weights) - making it x10+ as large. To mitigate, I figured two approaches:
- (1) Downsampling CNN output via bigger MaxPooling; downside: lowers effectiveness of stacked Bi-LSTMs, first of which has
return_sequences=True. Downsampling for Dense alone would create an information flow disparity
- (2) Inserting a narrower layer (e.g.
lstm_dim/2) before the
2*lstm_dimlayer; downside: (a) applies additional non-linearity to transformed features, possibly hampering residual learning
(b) The narrow layer may bottleneck the flow of information, with the wide layer oversampling the former and complicating learning - as with Autoencoders.
All considered, is a wider subsequent Dense layer unadvisable? Given the already-extensive hyperparameter space for my model, existing research / insight on the matter is appreciated.