Bayesian nonparametric answer to deep learning?

Question

As I understand it, deep neural networks are performing "representation learning" by layering features together. This allows learning very high dimensional structures in the features. Of course, it's a parametric model with fixed numbers of parameters, so it has the usual limitation that the model complexity may be difficult to tune.

Is there a Bayesian (nonparametric) way to learn such structures in the feature space, allowing the model complexity to adapt to the data? Related models include:

• Dirichlet processes mixture models, which allow one to partition the space into unlimited clusters, allowing the data to choose a finite number
• factorial models like Indian Buffet Process (IBP), which find potentially infinite number of latent features (aka topics) that explain the data.

However it seems that the IBP doesn't learn deep representations. There is also the issue that these methods are designed for unsupervised learning and usually we use deep learning for supervised tasks. Is there a variant of the IBP or other methods that allow representations to grow as the data demands?

Answer 1

As the other answer notes, a common non-parametric Bayesian alternative to neural networks is the Gaussian Process. (See also here).

However, the connection runs much deeper than that. Consider the class of models known as Bayesian Neural Networks (BNN). Such models are like regular deep neural networks except that each weight/parameter in the network has a probability distribution describing its value. A normal neural network is then somewhat like a special case of a BNN, except that the probability distribution on each weight is a Dirac Delta.

An interesting fact is that infinitely wide Bayesian neural networks become Gaussian Processes under some reasonable conditions.

Neal's thesis, Bayesian Learning for Neural Networks (1995) shows this in the case of a single-layer network with an IID prior. More recent work (see Lee et al, Deep Neural Networks as Gaussian Processes, 2018) extends this to deeper networks.

So perhaps you can consider large BNNs as approximations of a non-parametric Gaussian process model.

As for your question more generally, people often just need mappings in supervised learning, which it seems Bayesian non-parametrics are not as common for (at least for now), mostly for computational reasons (the same applies to BNNs, even with recent advances in variational inference). However, in unsupervised learning, they show up more often. For instance:

• Goyal et al, Nonparametric Variational Auto-encoders for Hierarchical Representation Learning, 2017

• Abbasnejad and Dick, Infinite Variational Autoencoder for Semi-Supervised Learning, 2017

• Chen, Deep Learning with Nonparametric Clustering, 2015

Answer 2

Hm I am not sure, but maybe deep gaussian processes might be one example of what you are looking for?

Deep Gaussian Processes

There is also more recent work on deep gaussian processes on scholar, but I am not knowledgeable enough to tell you what would be good to read:

https://scholar.google.de/scholar?as_ylo=2016&q=deep+gaussian+processes&hl=de&as_sdt=0,5&as_vis=1