3
$\begingroup$

Is anyone aware of research on neural networks tuning the existence of paths through their layers as part of their learning algorithm? Traditionally a neural net trains its weights and biases in order to improve performance on the cost function. I'm curious if there exist nets that train the existence of paths or connections - for example changing a layer into a fully connected layer, or subtracting paths to result in a non-fully connected layer.

In some sense this might be equivalent to setting weights along edges to 0 (subtracting paths). Since weights can be negative as well, a non-zero value supplies additional information beyond the existence of a path - it provides a magnitude (its absolute value) and a direction (its sign) as to the importance of the evidence it is weighing for the neuron it is feeding into.

So I'm curious if researchers have looked at path existence itself as a training parameter. I tried searching google and cross-validated for various permutations of this question but didn't come up with anything. I seem to remember a paper about selectively firing paths and learning connections but can't find it anymore. Pointers appreciated!

EDIT: Equivalently, this question might be posed as allowing the addition/subtraction of neurons in various layers (along with their connections). In this light it is not as simple as a reduction to a question of zero/non-zero values for weights.

Improve this question
$\endgroup$
4
  • $\begingroup$ As a partial answer to your question, have a look at ufldl.stanford.edu/tutorial/unsupervised/Autoencoders. In this Ng explains how an autoencoder can be regularized by penalizing neuron connections (coefficients) that are 'too large'. I believe that is 'sort of' what you are getting at. $\endgroup$
    – meh
    Oct 4, 2016 at 18:32
  • $\begingroup$ Can you define more clearly what a path is? $\endgroup$
    – bayerj
    Oct 4, 2016 at 18:57
  • $\begingroup$ @bayerj - i'm thinking of feedforward networks, so, a connection between a neuron in some layer l and some subsequent layer l+1. $\endgroup$
    – phoenixdown
    Oct 4, 2016 at 19:14
  • 1
    $\begingroup$ "optimal brain damage" (Lecun et al,1990), but I think "dropout" is what is actually being used nowadays $\endgroup$
    – seanv507
    Oct 4, 2016 at 22:59

1 Answer 1

Reset to default
1
$\begingroup$

Is this just the same as setting weight to zero?

Yes, setting weight to zero is equivalent to no connection.

Research on neural nets that learn paths/connections?

You may also want to look at the literature on neuro-evolution. Examples:

Equivalently, this question might be posed as allowing the addition/subtraction of neurons in various layers (along with their connections)

You can use Gaussian processes to determine this kind of hyperparameters. Example:

Improve this answer
$\endgroup$
2
  • $\begingroup$ Can you state how you think addition/subtraction of neurons is equivalent to setting weights to non-zero/zero values? $\endgroup$
    – phoenixdown
    Oct 4, 2016 at 22:11
  • 1
    $\begingroup$ @bobo Sorry I meant setting weight to zero is equivalent to no connection. Answer edited. $\endgroup$
    – Franck Dernoncourt
    Oct 4, 2016 at 22:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.