What type of neural network has feedback loops? The arrows in neural networks (see wikipedia image here) always seem to go from left to right. 
Is there a type of neural network where the hidden layer outputs connect to hidden layers inputs that are farther upstream?
 A: Yes, networks with skip-connections do that. But that's not a feedback loop, mind you, because it does not form a loop at all.
Here's the example from Wiki's article on Residual Neural Networks

See that 'Layer I-2' outputs are transmitted to 'Layer I' directly.
A: As a concrete example, imagine we have an input layer $I$, and layers $[I,Y]\rightarrow X\rightarrow Y\rightarrow X,Z$. In other words to compute $X$ we need both the input and the values from $Y$. As written, this is impossible to do since you do not yet have a value for $Y$ to plug into $X$. What you can do instead is use a placeholder of $0s$ (so if $Y$'s output is 100 dimensional, you'd have $(0,0,\cdots,0)$ 100 times) for $Y$ at the initial state, then compute $X$ and $Y$, and then recompute $X$ with a fresh value of $Y$. 
This is effectively the same structure as a recurrent neural network module $R$ that operates on the same input $X$, where $Y$ is the hidden state, e.g.:
$$Y\rightarrow R \rightarrow R\rightarrow R\rightarrow \cdots$$
$$\quad\mbox{ }\mbox{ }\mbox{ }\uparrow \quad\mbox{ }\uparrow\quad\mbox{ }\uparrow\cdots\quad$$
$$\mbox{ }\mbox{ }\mbox{ }I \quad\mbox{ }\mbox{ }\mbox{ }I\quad \mbox{ }\mbox{ }\mbox{ }\mbox{ } I \mbox{ }\cdots$$
The question now becomes when to stop, as technically you'd have to keep recomputing $X$. So the compromise here would be to set a maximum recurrence number, say 10, after which you stop the cycle and move onto $Z$ with your value of $Y$. 
