Does anyone in practice actually develop supervised model from scratch outside of classroom setting? I have a question in regards to why bother with developing a model from scratch and perform hyperparameter tuning when you can just use transfer learning for supervised learning.
The way that a machine learning model for supervised learning is developed is (provided that we have a good dataset),


*come up with some architecture

*train the model using first-order method

*validate using validation set

*tune the network to get good validation set performance

*test

And tuning the network involves adjusting the learning rate, batch-size, which is fine because these are model independent (no part of the model is affected).
But then there are also things like changing the number of hidden layers, number of neurons in each layer, and the choice of activation function, which is model dependent, because you are completely changing the model itself.
The question then becomes, since we are changing our model anyways, why not just grab an off-of-the-shelf model (such as VGG, GoogLeNet, etc.) which are known to perform well and start there to begin with, thereby potentially saving us a lot of work?
It doesn't seem logical for us to develop a model from scratch (as typically taught to students) which often times we have a feeling that it might not work so well, then gradually switch to more complicated architectures through a trial-and-error process of tuning, when you can directly start with the complicated architecture and do tuning on top of it.
Does anyone in practice actually develop model from scratch outside of classroom setting?
 A: This answer focuses mainly on the computer vision side of things. Also I guess I have a bit of a bias towards academic research practices.
It's probably uncommon to develop architectures completely from scratch, but there's also a wide spectrum between "from scratch" and "take an off-the-shelf model".
For example, you might start off your design with all the "common practices" -- relu, batch/instance norm, residual blocks, avg pooling, etc. This is pretty much "from scratch" as far as most people would consider it, but you're still pulling heavily from prior knowledge.
You could also incorporate architectural features known to be useful or helpful for the task at hand -- dilated / strided convs if a wide receptive field is needed, spectral norm if designing a discriminator, gated convs if the input isn't fully dense, U-net structure for raster outputs, etc. This is pretty far from designing "from scratch", but also pretty far from taking an off-the-shelf model.
Finally, a common design pattern in computer vision -- particularly object detection and segmentation -- is to have a big "backbone network" which extracts a feature map, followed by a number of "auxiliary networks" "branches" or "heads", which take the output of the backbone as their input and make the final task-specific predictions. Backbone networks include ResNet, ResNeXt, DenseNet, etc, and can often be easily swapped out for each other.

To finally get to your question: for "well-studied" problems such as classification, detection, and segmentation, the backbone approach is very common -- although this is not to say that the entire field consists of just tuning models -- there can be very interesting and novel questions about the design of the heads, which I wouldn't call tuning.
Stuff like generative models are typically much tricker to design and train, architectures can differ vastly depending on the target dataset, the model type (VAE vs GAN vs flows, etc), so "off-the-shelf" models don't work so great, unless you're simply trying to retrain the same model on very similar data. So there's definitely more "from scratch" design here.
Finally as Sycorax points out, there's many niche reasearch areas where designing new architectures from scratch (which hopefully perform better than currently known approaches) is the whole point!


It doesn't seem logical for us to develop a model from scratch (as
typically taught to students) which often times we have a feeling that
it might not work so well, then gradually switch to more complicated
architectures through a trial-and-error process of tuning, when you
can directly start with the complicated architecture and do tuning on
top of it.

Another perspective on this is that in a lot of cases, we're trying to investigate or improve on a secondary aspect of the network -- such as how the input / outputs are parameterized or preprocessed, or some data augmentation scheme, etc. It's common that the effect of these secondary aspects persists across many different network architectures, so it makes sense to start by experimenting with a simpler architecture (even if it doesn't perform as well), because it's faster to train / experiment with, and you can be more confident you won't run into any architecture specific idiosyncrasies (training instabilities, out of memory problems, uses batch norm which doesn't play nice with your objective function, etc). Then once you've made some progress, you just switch to using the better architecture, and confirm if your newfound knowledge / improvements also transfer over. And if they don't, that can be just as interesting from an academic perspective -- a possible sign of more interesting phenomena to investigate.
A: *

*Pre-trained networks don't always exist for your problem.
You've cited two common image classification networks, but it's unlikely these networks are at all meaningful for non-image problems. For example, I don't believe VGG would be successful on any tabular tasks, or something esoteric like interpreting a binary sequence with billions of elements.
I could even speculate that these are unlikely to succeed on unusual image tasks, like few- or one-shot recognition, image matching, or analysis of sonograms.

*

*Even restricted solely to image classification tasks, developing a model from scratch is useful.
The VGG and GoogLeNet models are the result of researchers working hard to figure out architectures which work. In turn, this means that reliance on pre-trained architectures is committing yourself to whatever shortcomings they possess. For example, a pre-trained model may be too expensive to use (e.g. have high latency, require too much memory) or perform poorly on your task.

The unstated assumption of this question is that all people training models have identical goals and constraints. This isn't true. Some people are trying to advance the state of the art by building a new model. Others are just trying to get a minimally viable model trained before a deadline. And so on. Deciding whether a pre-trained model could help achieve your goal is a common-sense way to attempt an efficient path forward.
