Suppose that we are tyring to infer the parametric distribution $p(y|\Theta(X))$, where $\Theta(X)$ is a vector output inverse [link function][1] with $\{\theta_1,\theta_2,...,\theta_M\}$.

We have a neural network at hand with some topology we decided. The number of outputs at the output layer match the number of parameters we would like to infer (it may be less if we don't care about all the parameters, as we will see in the examples below).

[![enter image description here][2]][2]

In the hidden layers we may use whatever activation function we like. What's crucial are the output activation function for each parameter as they have to be compatible with the support of the parameter.

[![enter image description here][3]][3]

Some example correspondence:

 - Linear activation: $\mu$, mean of Gaussian distribution
 - Logistic activation: $\mu$, mean of Bernoulli distribution
 - ReLU activation: $\sigma$, standard deviation of Gaussian distribution, shape parameters of Gamma distribution

Definition of cross entropy:

$$H(p,q) = -E_p[\log q(y)] = -\int p(y) \log q(y) dy$$

where $p$ is ideal truth, and $q$ is our model.

Empirical estimate:

$$H(p,q) \approx -\frac{1}{N}\sum_{i=1}^N \log q(y_i)$$

where $N$ is number of data points coming from $p$. 

Version for conditional distribution:

$$H(p,q) \approx -\frac{1}{N}\sum_{i=1}^N \log q(y_i|\Theta(X_i))$$

Now suppose that the network output is $\Theta(W,X_i)$ for a given input vector $X_i$ and all network weights $W$, then the training procedure for expected cross entropy is:

$$W_{opt} = \arg \min_W -\frac{1}{N}\sum_{i=1}^N \log q(y_i|\Theta(W,X_i))$$

Some examples:

 - Regression: Gaussian distribution with heteroscedasticity

$$\mu = \theta_1 : \text{linear activation}$$
$$\sigma = \theta_2: \text{ReLU activation}$$
$$\text{loss} = -\frac{1}{N}\sum_{i=1}^N \log \frac{1} {\theta_2(W,X_i)\sqrt{2\pi}}e^{-\frac{(y_i-\theta_1(W,X_i))^2}{2\theta_2(W,X_i)^2}}$$

under homoscedasticity we don't need $\theta_2$ as it doesn't affect the optimization and the expression simplifies to (after we throw away irrelevant constants):

$$\text{loss} = \frac{1}{N}\sum_{i=1}^N (y_i-\theta_1(W,X_i))^2$$

 - Binary classification: Bernoulli distribution

$$\mu = \theta_1 : \text{logistic activation}$$
$$\text{loss} = -\frac{1}{N}\sum_{i=1}^N \log [\theta_1(W,X_i)^{y_i}(1-\theta_1(W,X_i))^{(1-y_i)}]$$
$$= -\frac{1}{N}\sum_{i=1}^N y_i\log [\theta_1(W,X_i)] + (1-y_i)\log [1-\theta_1(W,X_i)]$$

with $y_i \in \{0,1\}$.

- Regression: Gamma response (never tried it before, but it should theoretically work)

$$\alpha \text{(shape)} = \theta_1 : \text{ReLU activation}$$
$$\beta \text{(rate)} = \theta_2: \text{ReLU activation}$$

$$\text{loss} = -\frac{1}{N}\sum_{i=1}^N \log [\frac{\theta_2(W,X_i)^{\theta_1(W,X_i)}}{\Gamma(\theta_1(W,X_i))} y_i^{\theta_1(W,X_i)-1}e^{-\theta_2(W,X_i)y_i}]$$

  [1]: https://en.wikipedia.org/wiki/Generalized_linear_model
  [2]: https://i.sstatic.net/1XxMq.jpg
  [3]: https://i.sstatic.net/kXXzb.png