A recent debate on so called Foundation models (CRFM) brings a real question of if we can build very large models on any specified domain, similar to current large language models, and replace our any statistical or machine learning modelling efforts to a tuning library of existing foundation models exercise. Obviously, causality can not be addressed by these models but this approach would change how we practice statistics and data science in general. Is this a new paradigm for statistics and machine learning?
Making the question a bit less opinion based as raised by the community. More specifically, given the proposed foundation models, do we have similar supervised pretraining (or broadly speaking transfer learning) practices from the statistics literature that fitted models are used as a starting point in new tasks?
Edit A recent article from Gradient's view Reflections on Foundation Models.
Edit Psychology Today has published a short article AI's Paradigm Shift to Foundation Models.
Edit A geophysics example Toward Foundation Models for Earth Monitoring: Proposal for a Climate Change Benchmark.
Edit A Generalist Agent from deepmind. Similar approach, "multi-domain learning".