I have just covered Artificial Neural Networks on Coursera's Machine Learning course and I would like to know more theory behind them. I find the motivation that they mimic biology somewhat unsatisfactory.

On the surface it appears that at each level we replace covariates with a linear combination of them. By doing it repeatedly we allow for non-linear model fitting. This begs the question: why the neural networks are sometimes preferred to just fitting a non-linear model.

More generally, I would like to know how Artificial Neural Networks fit within the Bayesian Framework of inference which is described in detail in E.T. Jaynes' book "Probability Theory: The Logic Of Science". Or, to put it simply, why do artificial neural networks work when they work? And, of course, the fact that they make successful predictions implies that they follow the aforementioned framework.


Here is a quote from "A Backward Look to the Future", by E. T. Jaynes.

New Adhockeries

In recent years the orthodox habit of inventing intuitive devices rather than appealing to any connected theoretical principles has been extended to new problems in a way that makes it appear at first that several new fields of science have been created. Yet all of them are concerned with reasoning from incomplete information; and we believe that we have theorems establishing that probability theory as logic is the general means of dealing with all such problems. We note three examples.

Fuzzy Sets are -- quite obviously, to anyone trained in Bayesian inference -- crude approximations to Bayesian prior probabilities. They were created only because their practitioners persisted in thinking of probability in terms of a "randomness" supposed to exist in Nature but never well defined; and so concluded that probability theory is not applicable to such problems. As soon as one recognizes probability as the general way to specify incomplete information, the reason for introducing Fuzzy Sets disappears.

Likewise, much of Artificial Intelligence (AI) is a collection of intuitive devices for reasoning from incomplete information which, like the older ones of orthodox statistics, are approximations to Bayesian methods and usable in some restricted class of problems; but which yield absurd conclusions when we try to apply them to problems outside that class. Again, its practitioners are caught in this only because they continue to think of probability as representing a physical "randomness" instead of incomplete information. In Bayesian inference all those results are contained automatically -- and rather trivially -- without any limitation to a restricted class of problems.

The great new development is Neural Nets, meaning a system of algorithms with the wonderful new property that they are, like the human brain, adaptive so that they can learn from past errors and correct themselves automatically (WOW! What a great new idea!). Indeed, we are not surprised to see that Neural Nets are actually highly useful in many applications; more so than Fuzzy Sets or AI. However, present neural nets have two practical shortcomings; (a) They yield an output determined by the present input plus the past training information. This output is really an estimate of the proper response, based on all the information at hand, but it gives no indication of its accuracy, and so it does not tell us how close we are to the goal (that is, how much more training is needed); (b) When nonlinear response is called for, one appeals to an internally stored standard "sigmoid" nonlinear function, which with various amplifications and linear mixtures can be made to approximate, to some degree, the true nonlinear function. (Note: emphasis mine.)

But, do we really need to point out that (1) Any procedure which is adaptive is, by definition, a means of taking into account incomplete information; (2) Bayes' theorem is precisely the mother of all adaptive procedures; the general rule for updating any state of knowledge to take account of new information; (3) When these problems are formulated in Bayesian terms, a single calculation automatically yields both the best estimate and its accuracy; (4) If nonlinearity is called for, Bayes' theorem automatically generates the exact nonlinear function called for by the problem, instead of trying to construct an approximation to it by another ad hoc device.

In other words, we contend that these are not new fields at all; only false starts. If one formulates all such problems by the standard Bayesian prescription, one has automatically all their useful results in improved form. The difficulties people seem to have in comprehending this are all examples of the same failure to conceptualize the relation between the abstract mathematics and the real world. As soon as we recognize that probabilities do not describe reality -- only our information about reality -- the gates are wide open to the optimal solution of problems of reasoning from that information.

A few comments:

  1. Point (a) ignores the developments in Bayesian Neural Networks, which started in the late eighties and early nineties (but notice that Jaynes's paper was written in 1993). Take a look at this post. Also, consider reading Yarin Gal's beautiful PhD thesis, and watching this great presentation by Zoubin Ghahramani.

  2. I don't see how point (b) could be a "shortcoming". In fact, it's the essence of why neural nets can approximate a large class of functions well. Notice that recent successful architectures moved from sigmoid to ReLU activations in the inner layers, favoring "depth" over "wideness". Approximation theorems have been recently proved for ReLU nets.

  • 2
    $\begingroup$ +1 Nothing is more satisfying than knowing exactly where one can find the precisely correct reference for an answer. $\endgroup$ – Sycorax Aug 1 '15 at 19:03
  • 5
    $\begingroup$ Given that the ad hoc devices demonstrated that they work in many situations, it would be productive to show (or disprove) that they simply fit into the Bayesian framework and, hence, gain a deeper understanding of the adhockeries, which are so widely deployed these days. This is the kind of work I am interested in. $\endgroup$ – Tom Artiom Fiodorov Aug 1 '15 at 19:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.