Research on neural nets that learn paths/connections? Is this just the same as setting weight to zero? Is anyone aware of research on neural networks tuning the existence of paths through their layers as part of their learning algorithm? Traditionally a neural net trains its weights and biases in order to improve performance on the cost function. I'm curious if there exist nets that train the existence of paths or connections - for example changing a layer into a fully connected layer, or subtracting paths to result in a non-fully connected layer.
In some sense this might be equivalent to setting weights along edges to 0 (subtracting paths). Since weights can be negative as well, a non-zero value supplies additional information beyond the existence of a path - it provides a magnitude (its absolute value) and a direction (its sign) as to the importance of the evidence it is weighing for the neuron it is feeding into.
So I'm curious if researchers have looked at path existence itself as a training parameter. I tried searching google and cross-validated for various permutations of this question but didn't come up with anything. I seem to remember a paper about selectively firing paths and learning connections but can't find it anymore. Pointers appreciated!
EDIT: Equivalently, this question might be posed as allowing the addition/subtraction of neurons in various layers (along with their connections). In this light it is not as simple as a reduction to a question of zero/non-zero values for weights.
 A: 
Is this just the same as setting weight to zero?

Yes, setting weight to zero is equivalent to no connection.

Research on neural nets that learn paths/connections?

You may also want to look at the  literature on neuro-evolution. Examples:


*

*Zaremba, Wojciech. Ilya Sutskever. Rafal Jozefowicz "An empirical exploration of recurrent network architectures." (2015): used evolutionary computation to find optimal RNN structures.

*Franck Dernoncourt. "The medial Reticular Formation: a neural substrate for action selection? An evaluation via evolutionary computation.". Master's Thesis. École Normale
Supérieure Ulm. 2011. used evolutionary computation to find optimal connections and number of neurons.

*Bayer, Justin, Daan Wierstra, Julian Togelius, and Jürgen Schmidhuber. "Evolving memory cell structures for sequence learning." In International Conference on Artificial Neural Networks, pp. 755-764. Springer Berlin Heidelberg, 2009.: used evolutionary computation to find optimal RNN structures.



Equivalently, this question might be posed as allowing the addition/subtraction of neurons in various layers (along with their connections)

You can use Gaussian processes to determine this kind of hyperparameters. Example:


*

*Franck Dernoncourt, Ji Young Lee Optimizing Neural Network Hyperparameters with Gaussian Processes for Dialog Act Classification, IEEE SLT 2016.

