Do models for artificial neural network growth, e. g. adaptive hidden layers, exist?

Question

Let's say we have a trained neural network that works. Training was performed with a set of given inputs and outputs (with training/validation division).

Now a new input node or output node is to be added to the ANN, increasing the input layer size from $j$ to $j+1$ or the output layer size to $l+1$. As the old training sets had a size of $j$ or $l$, respectively, they become useless in the new network, because they don't respect the information given by the new node (or the possible output for that node).

Me personally, I was thinking about "predefining" an ANN of the greatest possible and reasonable size, with -1 as input / output for unused nodes and no weight computing for those.

Now, is there a way to preserve the information that was stored in the old $j-k-l$ network into the new $j+1-k(+1)-l(+1)$ network? I.e., is there a way to "grow" an ANN without "forgetting" what it knew before it had its additional nodes?

I'm just trying to figure out how to handle the different input/output/weight vector sizes regarding the training/validation ones vs. those after adding nodes. I know that technically it's a "brand new ANN" after adding a node, but I figure there must be a workaround to preserve information. I'm using forward and backpropagation in matrix form, so it won't work out with differently sized vectors, and simply setting "zero" for unknown values is essentially wrong as I can't say "input zero" when I just don't know the input for a node that is "unknown" to me.

Me personally, I was thinking of predefining an ANN of the maximum possible and reasonable size, then setting the unused nodes to -1 and not computing any weights for them until used. But then again, how should the decision be made which hidden nodes to use, all of them? That would lead to huge amounts of virtually meaningless calculation and also overfitting. I wonder whether there is an alternative algorithm for selectively using only part of a hidden layer as needed.

Answer 1

I think it would be more practical to scale it up, so don't add a node, but double the number of nodes, and half the value of each connection. You're going to have to play around a bit with exactly how you implement this in order to get it acting normal, and then just mess with doubling the size and seeing how training the "half trained/half fresh" neural network affects the results.

As for an algorithm that only uses "Needed" nodes, that doesn't entirely make sense, because from a mathematical standpoint the algorithm is trying to match the training data as well as possible, and therefore every node "helps". Over-fitting isn't actually a failure of the backprop algorithm, it is when enough resources are provided that it can actually do it's job TOO well, and not actually recognize overarching patterns but specific input cases.