# Heterogenous layers in Neural Network

For this question when I say Neural Network I mean the standard Multi-layered Fully Connected Feed Forward Neural Network.

I'm exploring the different activation functions and cost functions used in Neural Networks. But given my limited experience with the subject (I'm only familiar with the $sigmoid$ activation and $mean$ $squared$ $error$ cost functions), I'm confused about the training of a Neural Network with heterogenous layers in terms of activation functions.

Questions:

1. Can a neural network have a combination of heterogenous layers as in the following examples: $$tanh \rightarrow tanh \rightarrow softmax$$ or $$ReLu \rightarrow ReLU \rightarrow sigmoid$$
2. If yes, what are the preferred combinations for classification tasks? and are the following gradient calculation and weight update rules right?

For the output layer: $$\delta_i = A'(z_i) \centerdot \frac{\delta C}{\delta a_i}$$ For the hidden layers: $$\delta_i = A'(z_i) \centerdot \sum_j{w_{ij}\delta_j} :$$ with $j$ is a unit in the next layer.

Weight update ($w_{ij}$ represents the weight from $i$th unit in a layer to the $j$th unit in the next layer): $$w_{ij} = w_{ij} + \eta \delta_j a_i$$

Notation used:

$\delta_i$ : the error at a given unit $i$.

$A$ : activation function.

$z_i$ : weighted input at a given unit $i$.

$C$ : the cost function.

$a_i$ : activation output at a given unit $i$.

$\eta$ : learning rate.