3
$\begingroup$

My CNN-LSTM EEG Keras classification model includes a Dense 'shortcut' connection for residual sequence learning as shown below; to match dimensionality, the Dense layer's set to 2*lstm_dim. When feeding very long sequences (>200,000 timesteps), this Dense layer dominates model size (# of weights) - making it x10+ as large. To mitigate, I figured two approaches:

  • (1) Downsampling CNN output via bigger MaxPooling; downside: lowers effectiveness of stacked Bi-LSTMs, first of which has return_sequences=True. Downsampling for Dense alone would create an information flow disparity
  • (2) Inserting a narrower layer (e.g. lstm_dim/2) before the 2*lstm_dim layer; downside: (a) applies additional non-linearity to transformed features, possibly hampering residual learning

(b) The narrow layer may bottleneck the flow of information, with the wide layer oversampling the former and complicating learning - as with Autoencoders.

All considered, is a wider subsequent Dense layer unadvisable? Given the already-extensive hyperparameter space for my model, existing research / insight on the matter is appreciated.

enter image description here

Improve this question
$\endgroup$
3
  • $\begingroup$ Did you find out answer to your question ? $\endgroup$
    – Daniel Wiczew
    Commented Jul 5, 2021 at 19:46
  • $\begingroup$ @DanielWiczew I'd say in general, it's a bad idea: we first bottleneck information then expect its recovery in an even more useful form. This can happen in only specific architectural settings, and with enough data. $\endgroup$
    – OverLordGoldDragon
    Commented Jul 5, 2021 at 20:12
  • $\begingroup$ I did a quick research yesterday and it's not always the case: arxiv.org/pdf/1803.00094.pdf In general wider layer after smaller layer is sometimes preferable. Try to play with playground.tensorflow.org and put 2:8:2 architecture. $\endgroup$
    – Daniel Wiczew
    Commented Jul 6, 2021 at 19:49

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.