Foundation models : Is it a new paradigm for statistics and machine learning?

Question

A recent debate on so called Foundation models (CRFM) brings a real question of if we can build very large models on any specified domain, similar to current large language models, and replace our any statistical or machine learning modelling efforts to a tuning library of existing foundation models exercise. Obviously, causality can not be addressed by these models but this approach would change how we practice statistics and data science in general. Is this a new paradigm for statistics and machine learning?

Edit

Making the question a bit less opinion based as raised by the community. More specifically, given the proposed foundation models, do we have similar supervised pretraining (or broadly speaking transfer learning) practices from the statistics literature that fitted models are used as a starting point in new tasks?

Edit A recent article from Gradient's view Reflections on Foundation Models.

Edit Psychology Today has published a short article AI's Paradigm Shift to Foundation Models.

Edit A geophysics example Toward Foundation Models for Earth Monitoring: Proposal for a Climate Change Benchmark.

Edit Along the similar lines, new architectures Google's PaLM and Open AI's DALL-E 2

Edit A Generalist Agent from deepmind. Similar approach, "multi-domain learning".

Edit ChatGPT: Optimizing Language Models for Dialogue

Edit This is probably quite significant work, applied in molecular physics and chemistry: A foundation model for atomistic materials chemistry (2023).

Answer 1

The Bitter Lesson is that in the long term, progress is dependent on leveraging more and more computational power. This is not to say that algorithmic and modeling progress isn't important, but they aren't the limiting factor -- neural networks have been since the 1950s (or earlier), and it's only now that increasing computation resources have let us exploit them fully.

The scaling hypothesis is the proposal that current models are only being held back by computation, and if we had several orders of magnitude more, we'd see dramatic improvements in modeling performance. This was explored and borne out by recent explorations into increasingly large language models.

(figure from here)

These recent large scale language models also demonstrate impressive few-shot or zero-shot capabilities, which validates the scaling hypothesis, and it sounds like the linked article concludes these "Foundation models" will come to replace more bespoke, individually trained models (although of course, no one is arguing that big models are going to replace the t-test).

Personally, I think there is a mountain of evidence for the bitter lesson, and for the scaling hypothesis, and these large language models are definitely very impressive. I don't have any opinion on whether this constitutes a new "paradigm" though (ideas like "the bitter lesson" have been floating around for many years, although the exploitation of supervised pretraining is relatively new), or whether these models will replace all others in the near future.

Answer 2

Disclosure: I didn't have time to carefully read the full paper yet.

From the abstract:

Despite the impending widespread deployment of foundation models, we currently lack a clear understanding of how they work, when they fail, and what they are even capable of due to their emergent properties.

Foundational models is a term with a good hype potential, but in reality there is nothing "foundational" in those models.

• The big language models can be fine-tunned or can serve as pre-trained layers of other models. It is not that they "replace" the other models in NLP. They don't solve "all" the problems.
• It is also not true that they are widespread. One thing is marketing and the companies that are trying to sell you those models, or sell you the costly infrastructure to run them. On another hand, a lot of problems is still sold with good old "shallow" machine learning, things like naive Bayes, rule-based algorithms, etc.
• There are results showing that "good old" models often outperform the big "SOTA" models, but nobody bothers to benchmark them anymore, but instead we go with the hype. One example are LSTMs.
• The big "foundational" models often fail miserably. When asked to generate text, they can produce complete nonsense (again, don't trust the cherry-picked marketing materials). One of such recent examples was the code generating GitHub Copilot by OpenAI, that was hugely overhyped, but in the end we learned that in 40% of cases it produces buggy code.
• Finally, the big NLP models exist mostly for English language. Good luck with finding equal quality models for other languages.

For me, the whole idea of "foundational models" is just "artificial general intelligence" idea in disguise. A lot of people still dream of building AGI, this didn't succeed, so now they're saying "Ok, let's make it almost-general". The problem is that we aren't there yet and we don't even know if we would ever reach it. At the same time, most of the real life problems are solved with relatively simple models, because they just work, are faster, cheaper, and usually easier to maintain.