# Behaviour of intralayer (lateral) connections in neural network during training

I am interested in experimenting with some very simple neural network architectures by building them in numpy.

I have followed a course in how to build a feed-forward multilayer neural network in numpy (e.g. Andrew Ng's brilliant course on Coursera), however, I am wondering what would happen if I were to add some intralayer (lateral) connections within one of the layers. I'd like to continue building my intuition in numpy, and so I am wondering how this might work on the forward pass and backward pass mathematically.

I understand and can implement the forward and backward passes for the fully connected network. And I understand that to introduce dropout the relevant column in the interlayer adjacency matrix needs to be zeroed. However, the intralayer adjacency matrix does not feature in this process as (usually) all the weights are 0.

My intuition tells me that the lateral connections should be treated the same as the interlayer connections.

In my example below, would the forward sequence go:

With the reverse sequence on the backward step? In this case, I guess this would look like creating a copy of the hiden layer, with only a connection between the 3rd and 4th node.

Or I could be completely wrong!

Edited to include modified diagram as per Sycorax's suggestion:

• What you're calling an "intra-layer" connection can be re-written so that the network still has strictly left-to-right flow for the forward pass, and uses skip connections; you just pin most of the weights to 0 in some layers. Since from this perspective, your proposed network is just a special case of a FFN with more than 1 hidden layer, the most common approach would be to just make the network deeper and let ordinary backprop update the weights in the usual way. But I guess you could include binary masks to enforce a certain kind of network connections, if you wish.
– Sycorax
Commented Aug 25, 2021 at 19:13
• Thanks - So in practice this would look like in steps: 1)input -> hidden layer (using connections between input and hidden layer) 2)hidden layer -> hidden layer (using only that one connection) 3)hidden layer -> output? I had a hunch this would be the right thing to do but clearly did not explain myself as well as I could have done. Commented Aug 25, 2021 at 19:15
• To depict this visually, all you would do is move your bottom weight in your diagram to the left, so that comprises its own hidden layer. The rest of the network works the same; you just need to include masks and skip connections to enforce the connections you depict. But, like I said, this is just a special case of a FFN with 2 hidden layers, which is why it's rarely seen. (Your network is not fully-connected in its current depiction; unclear if that's intended.)
– Sycorax
Commented Aug 25, 2021 at 19:20
• I've edited the diagram - is that what you meant? Commented Aug 25, 2021 at 19:30
• Yeah, that's exactly what I mean. Note that you also have skip connections from input to hidden layer 2.
– Sycorax
Commented Aug 25, 2021 at 19:38