Optimizing hyperparameters of network with extremely long training time As an example, let's say i am using a very deep fully convolutional autoencoder to segment lung scans. Input image resolutions will be large, since the features i hope to segment (things like early stage tumors) will be small and detail is important. In addition, lots of noise means lots of layers. Training time will be excessive.
When accuracy is critical, so is hyperparameter tuning. There are methods like gaussian optimization that will be much faster than random or grid searches, but even training the full model a single time will be expensive - gaussian optimization will require training several times.
My question is, what methodology or "best practice" do we have for situations like this? I read this paper about scaling CNNs, and from what i gathered, you can optimize a less complex model, procedurally scale the model until it is at the complexity you require, and it will be close to optimized. Use a gaussian optimizer once or twice to optimize the complex model further, and your accuracy very well may be superior. 
Although this is a large ask, i like to think it is very important, and this post could be an extremely useful resource. Many of my associates have encountered this roadblock, and i have not seen a single educational resource that addresses it. 
 A: Typically in architecture search or hyperparameter search problems, people use a proxy for "performance after a full training run" -- for example, you might measure the performance after training for only $K$ steps (where $K$ is small compared to a full training run). You might train a meta model to predict the performance of various hyperparameter configurations. If you're trying to select the best activation function, you might train smaller models with half as many layers to gauge the performance, etc. 
A: Halving search starts with a pool of $K$ candidates and reduces the pool by some fraction (nominally "half," but can be any fraction less than 1) at each iteration. So instead of training all $K$ candidates for $N$ epochs, you might start by training all $K$ candidates for 1 epoch, then take the top half of those candidates, and train the top half of candidates for 1 more epoch, and so on, reducing the total pool of candidates at each step while training each candidate model more.
Likewise, you can increase the amount of data you use for training at each iteration, so that each epoch proceeds more quickly at the start of training. Notionally, this might look like using 10% of training data for the first iteration, 20% of training data for the second iteration, and so on until you're training models using all of the data.
The overall idea is that there will be a number of candidate models that are rather poor, so we can move quickly to identify & discard them. So the premise of this method is that a model which does well (poorly) at the start of training also does well (poorly) after completing all of the training process. (And, in the alternative, if you don't believe that this is true for some reason, then you shouldn't use this method.)
A: If you are not altering network architecture try population based training. Otherwise consider bayesian optimization but limit trials to validation scores based on a subset of an epoch.
Assuming you have already sped up training by maxing out batch size via a dataloader (pytorch), preprocessed your data, and placed it into ram.
