1
$\begingroup$

I am interested in experimenting with some very simple neural network architectures by building them in numpy.

I have followed a course in how to build a feed-forward multilayer neural network in numpy (e.g. Andrew Ng's brilliant course on Coursera), however, I am wondering what would happen if I were to add some intralayer (lateral) connections within one of the layers. I'd like to continue building my intuition in numpy, and so I am wondering how this might work on the forward pass and backward pass mathematically.

I understand and can implement the forward and backward passes for the fully connected network. And I understand that to introduce dropout the relevant column in the interlayer adjacency matrix needs to be zeroed. However, the intralayer adjacency matrix does not feature in this process as (usually) all the weights are 0.

My intuition tells me that the lateral connections should be treated the same as the interlayer connections.

In my example below, would the forward sequence go:

  1. adjust input-hidden weight
  2. adjust hidden lateral weights
  3. adjust hidden-output weights

With the reverse sequence on the backward step? In this case, I guess this would look like creating a copy of the hiden layer, with only a connection between the 3rd and 4th node.

Or I could be completely wrong!

network with lateral connection

Edited to include modified diagram as per Sycorax's suggestion:

network with skip connection

$\endgroup$
6
  • $\begingroup$ What you're calling an "intra-layer" connection can be re-written so that the network still has strictly left-to-right flow for the forward pass, and uses skip connections; you just pin most of the weights to 0 in some layers. Since from this perspective, your proposed network is just a special case of a FFN with more than 1 hidden layer, the most common approach would be to just make the network deeper and let ordinary backprop update the weights in the usual way. But I guess you could include binary masks to enforce a certain kind of network connections, if you wish. $\endgroup$
    – Sycorax
    Aug 25 at 19:13
  • $\begingroup$ Thanks - So in practice this would look like in steps: 1)input -> hidden layer (using connections between input and hidden layer) 2)hidden layer -> hidden layer (using only that one connection) 3)hidden layer -> output? I had a hunch this would be the right thing to do but clearly did not explain myself as well as I could have done. $\endgroup$
    – Geodatsci
    Aug 25 at 19:15
  • $\begingroup$ To depict this visually, all you would do is move your bottom weight in your diagram to the left, so that comprises its own hidden layer. The rest of the network works the same; you just need to include masks and skip connections to enforce the connections you depict. But, like I said, this is just a special case of a FFN with 2 hidden layers, which is why it's rarely seen. (Your network is not fully-connected in its current depiction; unclear if that's intended.) $\endgroup$
    – Sycorax
    Aug 25 at 19:20
  • $\begingroup$ I've edited the diagram - is that what you meant? $\endgroup$
    – Geodatsci
    Aug 25 at 19:30
  • $\begingroup$ Yeah, that's exactly what I mean. Note that you also have skip connections from input to hidden layer 2. $\endgroup$
    – Sycorax
    Aug 25 at 19:38
1
$\begingroup$

What you're calling an "intra-layer" connection can be re-written so that the network still has strictly left-to-right flow for the forward pass, and uses skip connections; you just pin most of the weights to 0 in some layers. Since from this perspective, your proposed network is just a special case of a FFN with more than 1 hidden layer, the most common approach would be to just make the network deeper and let ordinary backprop update the weights in the usual way. But I guess you could include binary masks to enforce a certain kind of network connections, if you wish.

The second diagram that you've included shows that there's 2 different hidden layers, and several skip connections (from input to hidden 2, or hidden 1 to output). This makes it clear that this is a special kind of FFN, with a constrained kind of wiring.

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.