A recent debate on so called Foundation models (CRFM) brings a real question of if we can build very large models on any specified domain, similar to current large language models, and replace our any statistical or machine learning modelling efforts to a tuning library of existing foundation models exercise. Obviously, causality can not be addressed by these models but this approach would change how we practice statistics and data science in general. Is this a new paradigm for statistics and machine learning?


Making the question a bit less opinion based as raised by the community. More specifically, given the proposed foundation models, do we have similar supervised pretraining (or broadly speaking transfer learning) practices from the statistics literature that fitted models are used as a starting point in new tasks?

Edit A recent article from Gradient's view Reflections on Foundation Models.

Edit Psychology Today has published a short article AI's Paradigm Shift to Foundation Models.

Edit A geophysics example Toward Foundation Models for Earth Monitoring: Proposal for a Climate Change Benchmark.

Edit Along the similar lines, new architectures Google's PaLM and Open AI's DALL-E 2

Edit A Generalist Agent from deepmind. Similar approach, "multi-domain learning".

Edit ChatGPT: Optimizing Language Models for Dialogue

  • 5
    $\begingroup$ Interesting looking paper, I will give it a read. The abstract suggests it is trying to make the difficulty of ab-inito end-to-end training of deep neural networks a feature rather than a bug ;o) Deep learning is not a good solution to many machine learning problems, just as we should always try a linear model rather than (say) a non-linear SVM, we should try simpler, more transparent, ML tools before deep learning (IMHO). $\endgroup$ Sep 17, 2021 at 7:41
  • 5
    $\begingroup$ There is also an element of "if your only tool is a hammer, every problem looks like a nail". You can hammer in a screw, it will sort of work, but we should have screwdrivers in our toolboxes as well (and lots of other basic tools). $\endgroup$ Sep 17, 2021 at 7:50
  • 4
    $\begingroup$ That author list tho... $\endgroup$
    – qwr
    Sep 17, 2021 at 18:02
  • 1
    $\begingroup$ @DikranMarsupial sometimes the "simpler" models/tool hardly exist outside deep learning. For example, what (automatic, end-to-end) model is simpler to perform one of automatic translation, image classification & segmentation, or image & video captioning? If you don't have substantial training data, compute and time, pre-trained models are your best bet in obtaining an automatic system for each of these, and many other applications. $\endgroup$
    – Firebug
    Sep 23, 2021 at 12:19
  • 2
    $\begingroup$ This only may work in domains with unlimited amount of data available and where the relationships are stable. Example: trading on exchanges, where the data is in abundance, but relationships are elusive, so nothing works. Another: economic forecasting, where there is very little data, and nothing works. So this view is skewed by limited success in perceptive and physical domains, and the approach will not generalize to any domain $\endgroup$
    – Aksakal
    Nov 13, 2021 at 13:48

2 Answers 2


The Bitter Lesson is that in the long term, progress is dependent on leveraging more and more computational power. This is not to say that algorithmic and modeling progress isn't important, but they aren't the limiting factor -- neural networks have been since the 1950s (or earlier), and it's only now that increasing computation resources have let us exploit them fully.

The scaling hypothesis is the proposal that current models are only being held back by computation, and if we had several orders of magnitude more, we'd see dramatic improvements in modeling performance. This was explored and borne out by recent explorations into increasingly large language models.

enter image description here

(figure from here)

These recent large scale language models also demonstrate impressive few-shot or zero-shot capabilities, which validates the scaling hypothesis, and it sounds like the linked article concludes these "Foundation models" will come to replace more bespoke, individually trained models (although of course, no one is arguing that big models are going to replace the t-test).

Personally, I think there is a mountain of evidence for the bitter lesson, and for the scaling hypothesis, and these large language models are definitely very impressive. I don't have any opinion on whether this constitutes a new "paradigm" though (ideas like "the bitter lesson" have been floating around for many years, although the exploitation of supervised pretraining is relatively new), or whether these models will replace all others in the near future.

  • 1
    $\begingroup$ (+1) Scaling hypothesis and bitter lessons are directly relevant. Added a secondary question as well, if there were a prior "supervised pretraining" practice or similar from statistics literature. $\endgroup$ Sep 19, 2021 at 20:03
  • 1
    $\begingroup$ Some algorithms have improved much faster than compute power. Software is just hardware replacement and vice-versa. [1] Y. Sherry and N. C. Thompson, “How Fast Do Algorithms Improve?,” Proceedings of the IEEE, pp. 1–10, 2021, doi: 10.1109/JPROC.2021.3107219. $\endgroup$ Sep 22, 2021 at 11:53
  • 1
    $\begingroup$ The problem is that the scaling hypothesis only applies to problems that are compute limited, which is the sort of problem where deep learning has prospered as more compute power has become available, so those are the problems that get attention at the moment. But not all problems are compute limited, so we shouldn't be trying to present DL as a universal solution. It is just the sort of hype that has caused repeated slumps in neural network research since it's inception. Some problems require logic/reason/understanding rather than knowledge/intuition, and DL will not solve those alone. $\endgroup$ Sep 23, 2021 at 13:45
  • $\begingroup$ @PhilipOakley Nice citation. Quite relevant in this context. $\endgroup$ Sep 25, 2021 at 19:54

Disclosure: I didn't have time to carefully read the full paper yet.

From the abstract:

Despite the impending widespread deployment of foundation models, we currently lack a clear understanding of how they work, when they fail, and what they are even capable of due to their emergent properties.

Foundational models is a term with a good hype potential, but in reality there is nothing "foundational" in those models.

  • The big language models can be fine-tunned or can serve as pre-trained layers of other models. It is not that they "replace" the other models in NLP. They don't solve "all" the problems.
  • It is also not true that they are widespread. One thing is marketing and the companies that are trying to sell you those models, or sell you the costly infrastructure to run them. On another hand, a lot of problems is still sold with good old "shallow" machine learning, things like naive Bayes, rule-based algorithms, etc.
  • There are results showing that "good old" models often outperform the big "SOTA" models, but nobody bothers to benchmark them anymore, but instead we go with the hype. One example are LSTMs.
  • The big "foundational" models often fail miserably. When asked to generate text, they can produce complete nonsense (again, don't trust the cherry-picked marketing materials). One of such recent examples was the code generating GitHub Copilot by OpenAI, that was hugely overhyped, but in the end we learned that in 40% of cases it produces buggy code.
  • Finally, the big NLP models exist mostly for English language. Good luck with finding equal quality models for other languages.

For me, the whole idea of "foundational models" is just "artificial general intelligence" idea in disguise. A lot of people still dream of building AGI, this didn't succeed, so now they're saying "Ok, let's make it almost-general". The problem is that we aren't there yet and we don't even know if we would ever reach it. At the same time, most of the real life problems are solved with relatively simple models, because they just work, are faster, cheaper, and usually easier to maintain.

  • 5
    $\begingroup$ +1 Having trained my first neural network some time around 1990, I have seen several waves of this sort of hype, it is ironic that we (as a field) don't seem to be able to learn from the observations ;o) $\endgroup$ Sep 17, 2021 at 7:49
  • 3
    $\begingroup$ @DikranMarsupial if we build models that can't learn, what can we expect from ourselves? ;) $\endgroup$
    – Tim
    Sep 17, 2021 at 7:50
  • 1
    $\begingroup$ @Tim +1 Interesting connection with AGI. $\endgroup$ Sep 17, 2021 at 11:47
  • 1
    $\begingroup$ I honestly think that your second to last point is irrelevant – I can't imagine using any smaller/other models, such as LSTMs, producing working code 60% of the time. As far as I can tell, the fact that one family of models can achieve groundbreaking results in so many different tasks is rather foundational. $\endgroup$
    – Numeri
    Sep 18, 2021 at 0:09
  • 1
    $\begingroup$ @NumerisaysReinstateMonica “so many tasks” where all the tasks are re-framed to “given a prompt, generate plausible continuation” without any common sense, logical reasoning, understanding. Sure, the results are impressive, but still to great degree not usable (e.g. racist chatbots) and are just a small subset of what we’d need. $\endgroup$
    – Tim
    Sep 18, 2021 at 6:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.