Biological neurons are much more sophisticated than standard ANNs. Basing mostly on received impulses, they can learn long correlation chains between e.g. visual cortex, motor system and motivation system of a human newborn. There are approaches to re-engineer neocortex with simple local rules, for example hierarchical temporal memory, but mainly focused on neurons searching for sparse patterns and predicting them.

The big question is what local rules are sufficient for such single neuron, which gets some inputs and can make some local predictions based on them.

It turns out we can construct inexpensive artificial neurons directly learning correlations of inputs and using them for predictions - by modeling their joint probability distribution as a linear combination of orthonormal functions $\rho(x)=\sum a_f f(x)$, adaptation of these coefficients can be just: $a_f=(1-\lambda) a_f +\lambda f(x)$ for observation $x$ and some learning rate $\lambda$. Such artificial neuron can e.g. predict each input basing on the remaining ones ($E_{f,i}$ is expected value: $f$ with with constant on $i$-th variable):

enter image description here

Here is some orthonormal basis for a pair of variables - $|C|=1$ functions independently model density for each variable, $C=\{1,2\}$ model corrections to density from their correlations, e.g. the upper-left function models that growing one variable grows or shrinks the second (next - concentrates/spreads):

enter image description here

Are correlation-based artificial neurons considered in literature?

Can they be useful for real-world problems?

How to build practical neural-networks from them? Shallow supervised learning would just feed label to one of inputs, then ask for its prediction basing on the remaining inputs ... but what network architecture to use to get deep-learning level of predictions from them?

Here is simple simulator allowing to play with this density estimation method.

The closest model I was suggested so far is cascade correlations (CC) - where there are added new neurons corresponding to correlations - with frozen weights due to complex dependencies.

The approach here uses single neuron working as entire CC network - its adding neurons corresponds to adding new type of correlation to the basis of considered functions $f$. Some advantages:

  1. Instead of heuristic modeling, it is based on $L^2$ optimization, which thanks to using orthonormal basis, allows to estimate coefficients independently: $a_f$ is just average of $f(x)$ over the sample.

    So in contrast to CC, we don't longer have to freeze anything - all coefficients can be continuously adapted, we can freely add/remove functions from the basis.

    Additionally, we can use a larger basis for modeling, and only a smaller one for actual prediction - which coefficients are significant, larger than their uncertainty (can be also estimated). Independence allows also to use these coefficients as inputs for further layers.

  2. It can naturally contain non-linear correlations, e.g. for 2 variables:

    • function (1,1) models linear correlation: increase of the first variable, corresponds to increase or decrease of the second,

    • function (1,2) has parabola in second variable: increase of first variable corresponds to squeeze or spreed of the second,

    • finally they create complete basis, so theoretically could recreate any joint probability distribution, also for a larger number of variables.


Are correlation-based artificial neurons considered in literature?

First thing that comes to mind when I see this are Hopfield nets, which are stochastic binary neural networks. Boltzmann machines are a subset of Hopfield nets, and RBMs (Restricted BMs) are a subset which was particularly popular in the early days of deep learning. The popularity mostly stems from the fact that there is an efficient way to train these models and then assembling them to form multilayer networks.

How to build practical neural-networks from them?

I'm not really sure if I understood your model, but I'd look into methods that were used to stack RBMs to make deeper networks.

These slides by Hinton contain nice explanation of RBMs and related concepts. Hinton also has a guide on training RBMs with more details.

  • $\begingroup$ Thank you, I will have to take a closer look at RBM approaches, however, representing problem as a Boltzmann distribution is a very indirect way. Here we can directly model correlations between observed inputs: estimating multidimensional probability distribution they come from as: rho(x) = sum_f a_f f(x), for orthornomal basis {f} getting simple adaptation rule like: a_f -> (1-l) a_f + l * f(x). Correlations for C subset of variables allow to conclude about any of them from the remaining |C|-1 variables. $\endgroup$ – Jarek Duda May 5 '18 at 10:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.