Heterogenous layers in Neural Network

Question

For this question when I say Neural Network I mean the standard Multi-layered Fully Connected Feed Forward Neural Network.

I'm exploring the different activation functions and cost functions used in Neural Networks. But given my limited experience with the subject (I'm only familiar with the $sigmoid$ activation and $mean$ $squared$ $error$ cost functions), I'm confused about the training of a Neural Network with heterogenous layers in terms of activation functions.

Questions:

1. Can a neural network have a combination of heterogenous layers as in the following examples: $$tanh \rightarrow tanh \rightarrow softmax $$ or $$ReLu \rightarrow ReLU \rightarrow sigmoid$$

2. If yes, what are the preferred combinations for classification tasks? and are the following gradient calculation and weight update rules right?

   For the output layer: $$\delta_i = A'(z_i) \centerdot \frac{\delta C}{\delta a_i}$$ For the hidden layers: $$\delta_i = A'(z_i) \centerdot \sum_j{w_{ij}\delta_j} :$$ with $j$ is a unit in the next layer.

   Weight update ($w_{ij}$ represents the weight from $i$th unit in a layer to the $j$th unit in the next layer): $$w_{ij} = w_{ij} + \eta \delta_j a_i$$

Notation used:

$\delta_i$ : the error at a given unit $i$.

$A$ : activation function.

$z_i$ : weighted input at a given unit $i$.

$C$ : the cost function.

$a_i$ : activation output at a given unit $i$.

$\eta$ : learning rate.

Answer 1

1. Yes.

2. It depends on your architecture. There is not really a hard rule for this. Generally I'd say ReLU is especially beneficial in quite deep setups as it avoids vanishing and exploding gradients compared to tanh or sigmoids. But if you e.g. use an LSTM, which avoids the vanishing gradient problem in the first place with gating, tanh is behaving fine. So you should always analyze this question in regards to a specific setup.

Your formulas seem correct, although the weight update should subtract the deltas product as we usually want a positive learning rate.