Dynamically adjusting NN architecture: inventing the unnecessary? I am starting out on my PhD journey, and the ultimate goal that I set before myself is developing ANNs that would monitor the environment they work in and dynamically adjust their architecture to the problem at hand. The obvious implication is temporality of data: if the data set is not continuous and doesn't change over time, why adjust at all?
The big question is: with the recent rise of deep learning, is it still a relevant topic? Do FFNNs stand a chance to find themselves a niche in concept drift problems?
I fear to overload the thread with too many a question, but this one isn't entirely off-topic: I am aware of RNNs, but I have limited (ok, none, or purely theoretical) experience with them; I believe dynamic architecture adaptation must be a relevant topic in the context of RNNs. The question is, has it already been answered, and will I be reinventing the wheel?
P.S. Cross-posted to MetaOptimize
 A: Cascade-Correlation Neural Networks adjust their structure by adding hidden nodes during the training process, so this may be a place to start. Most of the other work I've seen that automatically adjusts the number of layers, number of hidden nodes, etc, of a neural network use evolutionary algorithms. 
Unfortunately, this work is out of my area so I can't recommend any particular papers or references to help you get started. I can tell you that I haven't seen any work which tries to jointly optimize network structure and parameters simultaneously within the deep learning community. In fact, most deep learning architectures are based on greedily learning a single layer at a time, thus making even online learning of deep neural networks a rather untouched area (the work of Martens et al. on Hessian Free Optimization being a notable exception). 
A: Another reason to consider developing novel approaches to constructive neural networks (such as the CC algorithm @alto mentioned) is in applications outside of statistics. In particular, in theoretical neuroscience and cognitive science, constructive neural networks are often used because of a metaphorical similarity to development and neurogenesis. For an example of heavy use of cascade-correlation for this, take a look at publications of Thomas R. Shultz. Unfortunately, the cascade correlation approach is biological unrealistic and if you have a neuroscience bend it is worth to consider how new NNs with adjustable architecture could be used as better models of development and/or neurogenesis.
