What is Bayesian Deep Learning and how does it relate to traditional Bayesian statistics and to traditional Deep Learning?
What are the main concepts and mathematics involved? Could I say it's just non parametric bayesian statistics? What are its seminal works as well as its current main developments and applications?
PS: Bayesian Deep Learning is getting a lot of attention, see NIPS workshop.
Going off of your NIPS workshop link, Yee Whye Teh had a keynote speech at NIPS on Bayesian Deep Learning (video: https://www.youtube.com/watch?v=LVBvJsTr3rg, slides: http://csml.stats.ox.ac.uk/news/2017-12-08-ywteh-breiman-lecture/). I think at some point in the talk, Teh summarized Bayesian deep learning as applying the Bayesian framework to ideas from deep learning (like learning a posterior over the weights of a neural network), and deep Bayesian learning as applying ideas from deep learning to the Bayesian framework (like deep Gaussian processes or deep exponential families). There are of course ideas that straddle the line between the two concepts, like variational autoencoders. When most people say Bayesian deep learning, they usually mean either of the two, and that's reflected in the accepted papers at the workshop you linked (along with the workshop the previous year). While the ideas go back to Neal's work on Bayesian learning of neural networks in the 90's (http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.446.9306&rep=rep1&type=pdf), and there's been work over the years since then, probably one of the more important recent papers would be the original variational autoencoder paper (https://arxiv.org/pdf/1312.6114.pdf).
I would suggest that you first get a good grasp of what is the underlying probabilistic model in a traditional Bayesian Neural Network. In the following, some terms will be written with a boldface. Please, try googling those terms to find more detailed information. This is just a basic overview. I hope it helps.
Let's consider the case of regression in feedforward neural networks and establish some notation.
Let $(x_1,\dots,x_p) =: \left(z^{(0)}_1,\dots,z^{(0)}_{N_0}\right)$ denote the values of the predictors at the input layer. The values of the units in the inner layers will be denoted by $\left(z^{(\ell)}_1,\dots,z^{(\ell)}_{N_\ell}\right)$, for $\ell=1,\dots,L-1$. Finally, we have the output layer $(y_1,\dots,y_k) =:\left(z^{(L)}_1,\dots,z^{(L)}_{N_L}\right)$.
The weights and bias of unit $i$ at layer $\ell$ will be denoted by $w^{(\ell)}_{ij}$ and $b^{(\ell)}_i$, respectively, for $\ell=1,\dots,L$, $i=1\dots,N_\ell$, and $j=1,\dots,N_{\ell-1}$.
Let $g^{(\ell)}_i : \mathbb{R}^{N_{\ell-1}} \to \mathbb{R}$ be the activation function for unit $i$ at layer $\ell$, for $\ell=1,\dots,L$ and $i=1\dots,N_\ell$.
Commonly used activation functions are the logistic, ReLU (aka positive part), and tanh.
Now, for $\ell=1,\dots,L$, define the layer transition functions $$ G^{(\ell)} : \mathbb{R}^{N_{\ell-1}} \to \mathbb{R}^{N_\ell} : \left(z^{(\ell-1)}_1,\dots,z^{(\ell-1)}_{N_{\ell-1}} \right) \mapsto \left( z^{(\ell)}_1,\dots,z^{(\ell)}_{N_\ell} \right), $$ in which $$ z^{(\ell)}_i = g^{(\ell)}_i\!\left( \sum_{j=1}^{N_{\ell-1}} w^{(\ell)}_{ij} z^{(\ell-1)}_j + b^{(\ell)}_i\right), $$ for $i=1,\dots,N_{\ell}$.
Denoting the set of weights and biases of all units in all layers by $\theta$, that is $$ \theta = \left\{ w^{(\ell)}_{ij},b^{(\ell)}_i : \ell=1,\dots,L \,;\, i=1\dots,N_\ell \,;\, j=1,\dots,N_{\ell-1} \right\}, $$ our neural network is the family of functions $G_\theta : \mathbb{R}^p\to\mathbb{R}^k$ obtained by composition of the layer transition functions: $$ G_\theta = G^{(L)} \circ G^{(L-1)} \circ \dots \circ G^{(1)}. $$
There are no probabilities involved in the above description. The purpose of the original neural network business is function fitting.
The "deep" in Deep Learning stands for the existence of many inner layers in the neural networks under consideration.
Given a training set $\{ (\mathbf{x}_i,\mathbf{y}_i) \in \mathbb{R}^p\times\mathbb{R}^k : i = 1,\dots,n \}$, we try to minimize the objective function $$ \sum_{i=1}^n \lVert \mathbf{y}_i-G_\theta(\mathbf{x}_i) \rVert^2, $$ over $\theta$. For some vector of predictors $\mathbf{x}^*$ in the test set, the predicted response is simply $G_\hat{\theta}(\mathbf{x}^*)$, in which $\hat{\theta}$ is the solution found for the minimization problem. The golden standard for this minimization is backpropagation implemented by the TensorFlow library using the parallelization facilities available in modern GPU's (for your projects, check out the Keras interface). Also, there is now hardware available encapsulating these tasks (TPU's). Since the neural network is in general over parameterized, to avoid overfitting some form of regularization is added to the recipe, for instance summing a ridge like penalty to the objective function, or using dropout during training. Geoffrey Hinton (aka Deep Learning Godfather) and collaborators invented many of these things. Success stories of Deep Learning are everywhere.
Probabilities were introduced in the picture in the late 80's and early 90's with the proposal of a Gaussian likelihood $$ L_{\mathbf{x},\mathbf{y}}(\theta,\sigma^2)\propto \sigma^{-n} \exp\left(-\frac{1}{2\sigma^2} \sum_{i=1}^n \lVert \mathbf{y}_i-G_\theta(\mathbf{x}_i) \rVert^2\right), $$ and a simple (possibly simplistic) Gaussian prior, supposing a priori independence of all weights and biases in the network: $$ \pi(\theta,\sigma^2) \propto \exp\left( -\frac{1}{2\sigma_0^2} \sum_{\ell=1}^L \sum_{i=1}^{N_\ell} \left( \left(b^{(\ell)}_i\right)^2 + \sum_{j=1}^{N_{\ell-1}} \left(w^{(\ell)}_{ij}\right)^2 \right) \right) \times \pi(\sigma^2).$$
Therefore, the marginal priors for the weights and biases are normal distributions with zero mean and common variance $\sigma_0^2$. This original joint model can be made much more involved, with the trade-off of making inference harder.
Bayesian Deep Learning faces the difficult task of sampling from the corresponding posterior distribution. After this is accomplished, predictions are made naturally with the posterior predictive distribution, and the uncertainties involved in these predictions are fully quantified. The holy grail in Bayesian Deep Learning is the construction of an efficient and scalable solution. Many computational methods have been used in this quest: Metropolis-Hastings and Gibbs sampling, Hamiltonian Monte Carlo, and, more recently, Variational Inference.
Check out the NIPS conference videos for some success stories: http://bayesiandeeplearning.org/
