Let's say we have a trained neural network that works. Training was performed with a set of given inputs and outputs (with training/validation division).
Now a new input node or output node is to be added to the ANN, increasing the input layer size from $j$ to $j+1$ or the output layer size to $l+1$. As the old training sets had a size of $j$ or $l$, respectively, they become useless in the new network, because they don't respect the information given by the new node (or the possible output for that node).
Me personally, I was thinking about "predefining" an ANN of the greatest possible and reasonable size, with -1 as input / output for unused nodes and no weight computing for those.
Now, is there a way to preserve the information that was stored in the old $j-k-l$ network into the new $j+1-k(+1)-l(+1)$ network? I.e., is there a way to "grow" an ANN without "forgetting" what it knew before it had its additional nodes?
I'm just trying to figure out how to handle the different input/output/weight vector sizes regarding the training/validation ones vs. those after adding nodes. I know that technically it's a "brand new ANN" after adding a node, but I figure there must be a workaround to preserve information. I'm using forward and backpropagation in matrix form, so it won't work out with differently sized vectors, and simply setting "zero" for unknown values is essentially wrong as I can't say "input zero" when I just don't know the input for a node that is "unknown" to me.
Me personally, I was thinking of predefining an ANN of the maximum possible and reasonable size, then setting the unused nodes to -1 and not computing any weights for them until used. But then again, how should the decision be made which hidden nodes to use, all of them? That would lead to huge amounts of virtually meaningless calculation and also overfitting. I wonder whether there is an alternative algorithm for selectively using only part of a hidden layer as needed.