My CNN-LSTM EEG Keras classification model includes a Dense 'shortcut' connection for residual sequence learning as shown below; to match dimensionality, the Dense layer's set to 2*lstm_dim. When feeding very long sequences (>200,000 timesteps), this Dense layer dominates model size (# of weights) - making it x10+ as large. To mitigate, I figured two approaches:

  • (1) Downsampling CNN output via bigger MaxPooling; downside: lowers effectiveness of stacked Bi-LSTMs, first of which has return_sequences=True. Downsampling for Dense alone would create an information flow disparity
  • (2) Inserting a narrower layer (e.g. lstm_dim/2) before the 2*lstm_dim layer; downside: (a) applies additional non-linearity to transformed features, possibly hampering residual learning

(b) The narrow layer may bottleneck the flow of information, with the wide layer oversampling the former and complicating learning - as with Autoencoders.

All considered, is a wider subsequent Dense layer unadvisable? Given the already-extensive hyperparameter space for my model, existing research / insight on the matter is appreciated.

enter image description here


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.