On October 9, 2024, the Nobel Foundation announced the Nobel Prize in Physics 2024 with the following statement of merit:
for foundational discoveries and inventions that enable machine learning with artificial neural networks
My question is: How precisely did the works of J. Hopfield and G. E. Hinton enable ML with ANNs, in the sense that they made ML with ANNs possible?
(Originally posted in AI, with modifications)
The quotation "for foundational discoveries and inventions that enable machine learning with artificial neural networks" is just the summary of the larger statement. Since OP omitted the particulars of which research the Nobel Committee referenced, and does not describe why they might be mistaken, I'm going to recap the parts of the statement that pertain to OP's stated question.
A large section of the statement is dedicated to Hopfield's work in statistical mechanics, and how it relates to neural networks via Hinton's research on (restricted) Boltzmann machines. The statement articulates the view that "physics has contributed tools for the development of machine learning" due to Hopfield, Hinton and their roles developing Boltzmann machines.
Going a bit deeper, in 2010, Hinton published a paper that popularized (but did not invent) the unit, in the paper entitled "Rectified Linear Units Improve Restricted Boltzmann Machines." This was a critical step forward for training deeper networks, which characterize modern NNs. While relus are useful in their own right, the connection to RBMs makes it complementary to the Nobel committee's statement that focuses on RBMs. (Though the statement does not explicitly mention ReLUs.) The paper has over 25,000 citations.
Moreover, the statement credits Hinton’s later development of stacking RBMs in deeper architectures was another step down the road to modern deep neural networks. The paper "Deep boltzmann machines" has over 3900 citations.
The committee credits RBMs with being an early kind of generative model, as an ancestor of modern generative AI models. I think it's clear that the intention of the committee is to draw connections between physics and this year's interest in all things GenAI.
I think it's fair to say that Hopfield and Hinton are important to the development of modern neural networks, particularly generative models. I’m not aware of any authorship controversy around RBMs, and the enormous citation counts on the relevant papers provide ample evidence that the contributions are foundational (to generative methods specifically, but providing tools that facilitate deeper networks). That said, I'll leave aside the question of whether their achievements pertain to physics.
A different way to approach the summary statement "for foundational discoveries and inventions that enable machine learning with artificial neural networks" is to ask "Are Hopfield Networks and (R)BMs are necessary and sufficient to machine learning research with artificial neural networks?" This framing leaves out the mention of generative AI that is a feature of the larger Nobel Committee's statement, instead turning attention to the history of machine learning with NNs more generally.
This answer is in the negative, because ancestry of modern artificial neural networks extends back much farther than Hopfield Networks and (R)BMs. An example is McCulloch and Pitts' work on the perceptron (1943). As a result, modern dense, feedforward nets are often called multi-layer perceptrons. However, if you go back that far, then it becomes more challenging to draw a linkage between artificial neural networks and physics.
Moreover, perceptrons are discriminative, but not generative. So the question "are these methods foundational?" hinges crucially on answering "foundational to what?"
I appreciate Firebug delineating this alternative way to view the Nobel Committee's statement in a comment.
Of course, Hinton has made other meaningful contributions to NNs. These are farther afield than the specific areas of research cited in the Committee's statement, but they are "foundational discoveries and inventions that enable machine learning with artificial neural networks."
Hinton’s paper on backpropagation is foundational to neural networks ("Learning representations by back-propagating errors."). This is how nearly every, or perhaps even all, contemporary neural networks are trained. Training neural networks is what enables a person to get high-quality predictions from them, so I think it’s fair to say this work enabled neural networks as a machine learning algorithm. It has over 54,000 citations.
The paper "Imagenet classification with deep convolutional-neural-networks" is a key work in the field of computer vision. It has over 160,000 citations.
For a number of years, dropout was an extremely popular regularization method. It continues to be of some interest when developing novel regularization methods. The dropout paper has over 52,000 citations.
The t-sne method continues to be a popular visualization tool today. It has over 47,000 citations.
The paper "Layer Normalization" has over 12,500 citations, and deepens the exploration of the "splat normalization" methods (batch-normalization, instance norm, etc.).
In total, his many, many papers have more than 800,000 citations. He’s certainly a key figure in the field of neural networks.
The purpose of this question, and answers to this question, is not to debate the relative merits of the recipients of the award. It's solely to answer OP's question, which is whether Hopfield's and Hinton's contributions enable modern neural networks.
Whether one agrees with the committee's assessment, or even considers generative models as important developments — let alone whether it pertains to the discipline of physics — is a matter of personal preference. I suppose if the Physics Nobel Committee wants to grant the award to a non-Physicist, they would have to find some (possibly tenuous) connection between the honoree's research and physics. Hopfield Networks and RBMs' relationship to statistical physics is, apparently, the Committee's chosen way to make that connection.
But the factual matter is that Hopfield and Hinton did the specific work identified by the committee and many, many other researchers used those ideas as stepping stones to advance to modern neural networks, to include so-called "GenAI" tools like ChatGPT, etc.
Rumelhart, D., Hinton, G. & Williams, R. "Learning representations by back-propagating errors." Nature 323, 533–536 (1986). https://doi.org/10.1038/323533a0
Salakhutdinov, Ruslan, and Geoffrey Hinton. "Deep boltzmann machines." Artificial intelligence and statistics. PMLR, 2009.
Nair, Vinod, and Geoffrey E. Hinton. "Rectified linear units improve restricted boltzmann machines." Proceedings of the 27th international conference on machine learning (ICML-10). 2010.
Srivastava, Nitish, et al. "Dropout: a simple way to prevent neural networks from overfitting." The journal of machine learning research 15.1 (2014): 1929-1958.
Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. "Imagenet classification with deep convolutional neural networks." Advances in neural information processing systems 25 (2012).
Van der Maaten, Laurens, and Geoffrey Hinton. "Visualizing data using t-SNE." Journal of machine learning research 9.11 (2008).
Lei Ba, Jimmy, Jamie Ryan Kiros, and Geoffrey E. Hinton. "Layer normalization." ArXiv e-prints (2016): arXiv-1607.
McCulloch, Warren, and Walter Pitts. "A Logical Calculus of the Ideas Immanent in Nervous Activity (Bulletin of Mathematical Biophysics, vol. 5, 1943)." Online: https://doi.org/10.1007/BF02478259.
Backprop is perhaps the most foundational idea behind a lot of modern "AI". This image was posted recently on the CONNECTIONISTS mailing list by Prof. Asim Roy. It apparently shows Geoffrey Hinton at a panel discussion at ICNN-97 - This slide gives a list of the things wrong with backprop and item 5 is "Amari thought of it first" ;o)
Prof. Amari was apparently also taking part in the discussion. Not a full answer, but relevant. Also good nostalgia value for the nature of conferences in the 1980s and 1990s!
Hinton and Hopfield have done great work, including foundational work, the trouble is the foundational work has negligible physics component, and the bit with a physics component wasn't that foundational (to modern AI)
