As I understand it, deep neural networks are performing "representation learning" by layering features together. This allows learning very high dimensional structures in the features. Of course, it's a parametric model with fixed numbers of parameters, so it has the usual limitation that the model complexity may be difficult to tune.
Is there a Bayesian (nonparametric) way to learn such structures in the feature space, allowing the model complexity to adapt to the data? Related models include:
- Dirichlet processes mixture models, which allow one to partition the space into unlimited clusters, allowing the data to choose a finite number
- factorial models like Indian Buffet Process (IBP), which find potentially infinite number of latent features (aka topics) that explain the data.
However it seems that the IBP doesn't learn deep representations. There is also the issue that these methods are designed for unsupervised learning and usually we use deep learning for supervised tasks. Is there a variant of the IBP or other methods that allow representations to grow as the data demands?