In a 'typical' neural net that uses forward and backward propagation, we modify the weights to minimize the objective loss function. But what if I want to add nodes or layers instead? For example, consider an autoencoder network that is being trained for denoising. 'Typically' some voodoo is needed to select the layer sizes and the number of layers as well as the minimum number of nodes at the compressed representation.
Suppose instead,I want to start with 10 densely connected layers each with 2 nodes. The I want run 5 epochs of training. Then I want to decrease the number of layers. This event would define a 'super-epoch'. Then I would repeat the 5 epochs of training. If the loss function is better, then repeat with another layer decrease. If the loss function is worse, go back to the previous architecture and add a node. And repeat.
You might say this just kicks the can down the road. Instead of arbitrarily picking an architecture and optimizing it, I am now doing trial and error, trying to brute force an architecture in a random way. Perhaps for no improvement.
While backpropagation allows us to calculate the gradient, there is no easy analog to allow calculation of a 'gradient' from one architecture to the next. Or is there?
In any event, my attempt at writing such a 'scalable' network in python drowned my brain. I literally ran out of memory in my skull and on my pc. I have heard that pytorch allows this kind of thing (dynamical networks?)
