1
$\begingroup$

My CNN-LSTM EEG Keras classification model includes a Dense 'shortcut' connection for residual sequence learning as shown below; to match dimensionality, the Dense layer's set to 2*lstm_dim. When feeding very long sequences (>200,000 timesteps), this Dense layer dominates model size (# of weights) - making it x10+ as large. To mitigate, I figured two approaches:

  • (1) Downsampling CNN output via bigger MaxPooling; downside: lowers effectiveness of stacked Bi-LSTMs, first of which has return_sequences=True. Downsampling for Dense alone would create an information flow disparity
  • (2) Inserting a narrower layer (e.g. lstm_dim/2) before the 2*lstm_dim layer; downside: (a) applies additional non-linearity to transformed features, possibly hampering residual learning

(b) The narrow layer may bottleneck the flow of information, with the wide layer oversampling the former and complicating learning - as with Autoencoders.

All considered, is a wider subsequent Dense layer unadvisable? Given the already-extensive hyperparameter space for my model, existing research / insight on the matter is appreciated.


enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.