In a paper about deep neural networks they say:

In a general feedforward linear network described by an underlying directed acyclic graph, units can be organized into layers using the shortest path from the input units to the unit under consideration. The activity in unit $i$ of layer $h$ can be expressed as:

$$S^h_i(I)=\sum_{l<h}\sum_jw^{hl}_{ij}S^l_j \,\,\,\,\,\,\,\mathrm{with}\,S^0_j=I_j$$

I'm a little confused by the upper indexing on $w$. What would it mean if $w$ had an upper index $(h,h-2)$, for instance?


This formula describes the case when the units in one layer have not only connections to the immediately previous layers, but to all previous layers.

The weight $w^{hl}_{ij}$ is simply denoting the strength of the connection between the unit $i$ in layer $h$ and the unit $j$ in some previous layer $l$. In most feed-forward networks without any skip/residual/shortcut connections, $w^{h,h-2}$ will be zero.

  • $\begingroup$ This is taken from page 5 of the linked article. In the previous sentence (the first sentence of section 3), he says that which layer a unit A is in can be defined by the shortest path from the input layer to A. So doesn't this definition ensure that layer h could never be connected to layer $h-2$? $\endgroup$ – Eric Auld Feb 17 '18 at 15:14
  • $\begingroup$ I suppose the definition is erroneous and should be rather "the longest path". Or perhaps they consider a model without any skip connections and then shortest=longest, but in that case the formula is somehow incorrect. $\endgroup$ – Jan Kukacka Feb 17 '18 at 15:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.