# How to optimize the hyperparameters of a Deep Gaussian Process?

I am trying to understand an article from NIPS 2017 where Alaa and van der Schaar create a Deep Multitask Gaussian Processes (DMGP) with competing risks. I don't grab what are they trying to optimize.

In their section 4 they present their inference model :

Conducting the survival analysis rerquires computing the posterior probability density $d\mathbb P(T^*|D,X^*,\Theta_Z,\Theta_T)$ for a given out-o-sample point $X^*$ with $T^*=g(X^*)$. We follow and empirical Bayes approach for updating the posterior on $g$. Taht is, we first tune the hperparameters $\Theta_Z$ and $\Theta_T$ using the offline dataset $D$, and then for any out-of-sample patient with covariates $X^*$, we evaluate $d\mathbb P(T^*|D,X^*,\Theta_Z,\Theta_T)$ by direct Monte Carlo sampling.

They calibrate the hyperparameters by maximizing the marginal likelihood $d(\mathbb{P}|\Theta_Z,\Theta_T)$. For every subject $i$ in $D$, they observe a "label" of the form $(T_i,k_i)$ which indicates the type of event that occured along with the time of its occurence. Since $T_i$ is the smallest element in T, then the label $(T_i,k_i)$ is informative of all the events (i.e. all the learning tasks) in $K/\{k_i\}$; we know that $T_i^j\ge T_i\forall j\in K/\{k_i\}$ (I would have create a strict inequality as far as they said that two events can't occur at the same time). They also note that the subject's data may be right-censored, i.e. $k_i=\emptyset$, which implies that $T_i^j\ge T_i,\forall j\in K$. Hence, the likelihood of the survival information in $D$ is :

$$d\mathbb{P}(\{X_i,T_i,k_i\}_{i=1}^n|\Theta_Z,\Theta_T)\sim d\mathbb P(\{\mathcal T_i\}_{i=1}^n|\{X_i\}_{i=1}^n,\Theta_Z,\Theta_T\}$$

Where

\begin{align} (3):\{\mathcal T_i\}_{i=1}^n=\begin{cases}\{T_i^{k_i}=T_i,\{T_i^j\ge T_i\}_{j\in K/\{k_i\}}\}\mbox{ if } k_i\neq \emptyset\\\{T_i^j\ge T_i\}_{j\in K}\mbox{ if } k_i = \emptyset \end{cases} \end{align}

We can write the marginal likelihood in $(3)$ as the conditional density by marginalizing over the conditional distribution of the hidden variable $Z_i$ as follows :

\begin{align}(4): d\mathbb P(\{\mathcal T_i\}_{i=1}^n|\{X_i\}_{i=1}^n,\Theta_Z,\Theta_T)=\int \underbrace{d\mathbb P(\{\mathcal T_i\}_{i=1}^n |\{Z_i\}_{i=1}^n,\Theta_T)d\mathbb P(\{Z_i\}_{i=1}^n,\Theta_Z)}_{\mbox{Why a product ?}} \end{align}

Since the integral in $(4)$ is intractable, they follow the variational inference scheme proposed Damaniou and Lawrance's article on Deep Gaussian Processes where their tune the hyperparameters by maximizing the following variational bound on $(4)$ :

$$\mathcal F=\int_{Z,f_z,f_t}Q.\log(\frac{d\mathbb(\{\mathcal T_i\}_{i=1}^n,\{Z_i\}_{i=1}^n,\{f_z(X_i)\}_{i=1}^n,\{f_T(Z_i)\}_{i=1}^n|\{X_i\}_{i=1}^n,\Theta_Z,\Theta_T)}{Q})$$

Where $Q$ is a variational distribution, and $F\le \log d\mathbb P(\{\mathcal T_i\}_{i=1}^n|\{X_i\}_{i=1}^n,\Theta_Z,\Theta_T)$.

[...] They use the adaptrive moment arlgorithm (ADAM) to optimize $\mathcal F$ with respect to $\Theta$s.

I don't get this think. I thought we had to optimize the hyperparameters $\Theta$s but it seems we try to optimize the $\mathcal F$. But if we optimize $\mathcal F$ playing on $\Theta$s we modificate the $\log d\mathbb P(\{\mathcal T_i\}_{i=1}^n|\{X_i\}_{i=1}^n,\Theta_Z,\Theta_T)$ part.

I have not yet read the whole article Damaniou and Lawrance's.

I'm a research student but I am very slow to understand all the maths.

• Since the marginal likelihood is intractable, they're instead maximizing a lower bound of the marginal likelihood (the ELBO/variational lower bound, which they call script F). They're still optimizing with respect to the hyperparameters. – aleshing Dec 28 '17 at 2:36