How to design a Neural Network model that combines components of Feedforward and Recurrent features?

Question

I want to design an end-to-end system that has components of both feedforward neural networks and recurrent neural networks. For example the data can have different components (some sequential in nature, others not, perhaps images or even a single number that is better processed by a feedforward network).

My main concern is that when combining different components of different networks I might be combining things that work well on one but not as well on the other. Or perhaps one mainly has $tanh(x)$ as activations while the other has mainly $relu(x) = max(0,x)$. In these cases I am unsure if my model might suffer and not train or who knows what other issue might arise. Other concerns I have was that as far as I know some methods like batch-normalization work mostly for feedfoward models but not as well on RNNs. I know there have been many discoveries that can made DL work now days so I want to be able to combine them appropriately.

So my question is, when we mix components from different types of Neural Networks, are there any heuristics or known empirical results to how to combine these different blocks? e.g. as random ideas I have right now, if we use LSTMS combined with feedfoward nets perhaps we always need to have residual connections (that look similar to memory channels) or if we use RNNs + FFs then we always have to use ReLUs everywhere...or tanh's everywhere...Is perhaps the trick to insert normalization layers here and there? Or Selu's here and there?

How do we combine different type models successfully (especially recurrent nets with feedforward nets)?

---

An example task perhaps of this form could be the input X being an Image and a Question (text) to 5 options (multiple choice). Or Image and Question and we have to answer a question about it (e.g. CLEVR data set).

There are many problems of this sort.

---

Cross posted: https://qr.ae/TWvXri

Answer 1

I wouldn't worry so much about internal structural decisions like which activations to use - for these there is no "right answer", and you can just test multiple architectures with a hyperparameter search as normal. That said, it would probably be helpful to supplement your network with auxiliary outputs (if possible) to help train each input processing architecture in isolation. To use your example of the CLEVR dataset, you could include an auxiliary output to the natural language processing layers which tries to output a representation of the corresponding "functional program". Likewise, if there are some annotations of the image content, add these as an auxiliary output to the image processing layers.

Otherwise, the only major thing to get right is to process your inputs correctly so they make the most sense possible to the rest of the network. That means processing sequential data with a sequential layer, image data with convolution layer/s etc. You can then concatenate different inputs and pass them through some dense layers as you wish.

Here's a simple example from the keras documentation which combines sequential test data with auxiliary (non-sequential) data. To once again use the example of the CLEVR dataset, you could:

• Replace this auxiliary input with a CNN giving a dense representation of the image
• Replace the LSTM auxiliary output with a representation of the functional program
• Add an auxiliary output to the CNN which outputs information about the content of the image (if this is available)