Artificial neural networks have a bad reputation of being a black box. More over in cases when we do have some prior knowledge about the domain of a particular supervised learning problem it is not obvious how to introduce it to the model.

On the other hand Bayesian models and the state of art of those - Bayesian networks - solve this problem naturally. But these models have their own known limitations.

  • Is it possible to take best from the both kinds of models. Are there any theory or practical success stories of combining both kinds of models into a some hybrid.

  • And, in general, what are the known strategies to incorporate prior knowledge into a neural network model (feed forward or recurrent)

  • $\begingroup$ would transfer learning count as prior knowledge for you? $\endgroup$ – Aksakal Mar 19 '18 at 13:04

Actually, there are many ways to incorporate prior knowledge into neural networks. The simplest type of prior knowledge often used is weight decay. Weight decay assumes the weights come from a normal distribution with zero mean and some fixed variance. This type of prior is added as an extra term to the loss function, having the form:

$$\mathcal{L}(w) = E(w) + \lambda\frac{1}{2}||w||_2^2,$$

where $E(w)$ is the data term (e.g. a MSE loss) and $\lambda$ controls the relative importance of the two terms; it is also proportional to the prior variance. This corresponds to the negative log-likelihood of the following probability: $$p(w|\mathcal{D})\propto p(\mathcal D|w)p(w),$$ where $p(w)=\mathcal N(w|0,\lambda^{-1}I)$ and $-\log p(w)\propto -\log\,\exp(-\frac{\lambda}{2}||w||_2^2)=\frac{\lambda}{2}||w||_2^2$. This is the same as the bayesian approach to modeling prior knowledge.

However, there are also other, less straight-forward methods to incorporate prior knowledge into neural networks. They are very important: prior knowledge is what really bridges the gap between huge neural networks and (relatively) small datasets. Some examples are:

Data augmentation: By training the network on data perturbed by various class-preserving transformations, you are incorporating your prior knowledge about the domain, namely the transformations that the network should be invariant to.

Network architecture: One of the most successful neural network techniques of the past decades are the convolutional networks. Their architecture sharing limited field-of-view kernels over spatial locations brilliantly exploits our knowledge about data in image space. This is also a form of prior knowledge incorporated into the model.

Regularization loss terms: Similar to weight decay, it is possible to construct other loss terms which penalize mappings contradicting our domain knowledge.

For an in-depth analysis/overview of these methods, I can point you to my article Regularization for Deep Learning: A Taxonomy. Also, I recommend looking into bayesian neural networks, meta-learning (finding meaningful prior information from other tasks in the same domain, see e.g. (Baxter, 2000)), possibly also one-shot learning (e.g. (Lake et al., 2015)).

  • $\begingroup$ (+1) This is a useful read. Could you expand on what you mean by 'perturbed by various class-preserving transformations'? I don't quite understand. $\endgroup$ – mkt - Reinstate Monica Mar 19 '18 at 13:01
  • 1
    $\begingroup$ Consider data augmentation e.g. by rotation or mirroring images. It changes the image, but its class (label) remain the same. The richer set of transformations you use, the more prior information you encode in the model (through the augmented dataset). $\endgroup$ – Jan Kukacka Mar 19 '18 at 13:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.