I am starting out on my PhD journey, and the ultimate goal that I set before myself is developing ANNs that would monitor the environment they work in and dynamically adjust their architecture to the problem at hand. The obvious implication is temporality of data: if the data set is not continuous and doesn't change over time, why adjust at all?

The big question is: with the recent rise of deep learning, is it still a relevant topic? Do FFNNs stand a chance to find themselves a niche in concept drift problems?

I fear to overload the thread with too many a question, but this one isn't entirely off-topic: I am aware of RNNs, but I have limited (ok, none, or purely theoretical) experience with them; I believe dynamic architecture adaptation must be a relevant topic in the context of RNNs. The question is, has it already been answered, and will I be reinventing the wheel?

P.S. Cross-posted to MetaOptimize

  • $\begingroup$ When you say "adjust their architecture" do you mean the parameters (weights, biases) or updating the actual structure of the network (hidden nodes, activation function, connectivity, etc)? Also, in many deep learning applications the end result IS a feed forward neural network, just one with weights initialized by some unsupervised process. $\endgroup$ – alto Aug 9 '12 at 15:04
  • $\begingroup$ @alto, I am referring to the actual NN structure - number of hidden units and (possibly) layers - I am sure it can be implemented at different levels of complexity. I feel that I have to start reading up on deep learning if I am to get anywhere at all. $\endgroup$ – anna-earwen Aug 10 '12 at 12:03
  • $\begingroup$ @anna-earwen interesting PhD topic, how is it going, any publications yet? $\endgroup$ – Dikran Marsupial May 12 '14 at 12:03
  • 1
    $\begingroup$ @Dikran Marsupial, I will soon head to IJCNN 2014 to talk about how and why PSO fails to train high-dimensional NNs. So the answer is yes and know: I took a big detour from the original research vector, and I wonder if I will still come back to the adjustible architectures. Only time and empirical results will tell! $\endgroup$ – anna-earwen May 14 '14 at 12:24
  • $\begingroup$ I will look out for it in the proceedings - understanding why things don't work is something science needs more of (and solid empirical studies). $\endgroup$ – Dikran Marsupial May 14 '14 at 17:56

Cascade-Correlation Neural Networks adjust their structure by adding hidden nodes during the training process, so this may be a place to start. Most of the other work I've seen that automatically adjusts the number of layers, number of hidden nodes, etc, of a neural network use evolutionary algorithms.

Unfortunately, this work is out of my area so I can't recommend any particular papers or references to help you get started. I can tell you that I haven't seen any work which tries to jointly optimize network structure and parameters simultaneously within the deep learning community. In fact, most deep learning architectures are based on greedily learning a single layer at a time, thus making even online learning of deep neural networks a rather untouched area (the work of Martens et al. on Hessian Free Optimization being a notable exception).

  • $\begingroup$ Thanks a lot, you already gave me enough information to start digging for gold. :) $\endgroup$ – anna-earwen Aug 10 '12 at 20:09

Another reason to consider developing novel approaches to constructive neural networks (such as the CC algorithm @alto mentioned) is in applications outside of statistics. In particular, in theoretical neuroscience and cognitive science, constructive neural networks are often used because of a metaphorical similarity to development and neurogenesis. For an example of heavy use of cascade-correlation for this, take a look at publications of Thomas R. Shultz. Unfortunately, the cascade correlation approach is biological unrealistic and if you have a neuroscience bend it is worth to consider how new NNs with adjustable architecture could be used as better models of development and/or neurogenesis.

  • 1
    $\begingroup$ Thanks, Artem! In fact, I am more of a pure computer scientist than anything else, thus my knowledge of neuro- and congnitive science is less than scarce. Sounds exciting, though, and since all roads are still open, I could delve into this, too - at least to some extent. At the moment I am particularly interested in applications to real-life engineering and data analysis problems that could work for benchmarking. $\endgroup$ – anna-earwen Aug 16 '12 at 19:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.