0
$\begingroup$

The arrows in neural networks (see wikipedia image here) always seem to go from left to right.

Is there a type of neural network where the hidden layer outputs connect to hidden layers inputs that are farther upstream?

$\endgroup$
  • $\begingroup$ The title of the question is confusing: it mentions feedback loops (In which case the answer is straightforward - you are looking for RNN) , but then in the text of the post what you describe is something different from feedback loops, and it seems like you are looking for ELM or ESN networks (Hopfiled might fit the bill as well). $\endgroup$ – Skander H. Jul 8 '19 at 2:23
0
$\begingroup$

As a concrete example, imagine we have an input layer $I$, and layers $[I,Y]\rightarrow X\rightarrow Y\rightarrow X,Z$. In other words to compute $X$ we need both the input and the values from $Y$. As written, this is impossible to do since you do not yet have a value for $Y$ to plug into $X$. What you can do instead is use a placeholder of $0s$ (so if $Y$'s output is 100 dimensional, you'd have $(0,0,\cdots,0)$ 100 times) for $Y$ at the initial state, then compute $X$ and $Y$, and then recompute $X$ with a fresh value of $Y$.

This is effectively the same structure as a recurrent neural network module $R$ that operates on the same input $X$, where $Y$ is the hidden state, e.g.:

$$Y\rightarrow R \rightarrow R\rightarrow R\rightarrow \cdots$$ $$\quad\mbox{ }\mbox{ }\mbox{ }\uparrow \quad\mbox{ }\uparrow\quad\mbox{ }\uparrow\cdots\quad$$ $$\mbox{ }\mbox{ }\mbox{ }I \quad\mbox{ }\mbox{ }\mbox{ }I\quad \mbox{ }\mbox{ }\mbox{ }\mbox{ } I \mbox{ }\cdots$$

The question now becomes when to stop, as technically you'd have to keep recomputing $X$. So the compromise here would be to set a maximum recurrence number, say 10, after which you stop the cycle and move onto $Z$ with your value of $Y$.

$\endgroup$
2
$\begingroup$

Yes, networks with skip-connections do that. But that's not a feedback loop, mind you, because it does not form a loop at all.

Here's the example from Wiki's article on Residual Neural Networks

enter image description here

See that 'Layer I-2' outputs are transmitted to 'Layer I' directly.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.