I was wondering if the different notations of the softmax input mean different things especially about the size of the output. For example, in the paper Pointer Networks, it sometimes state the input of the softmax with the subscript j,

Equation with j

and in another equation, it removed the subscript

enter image description here

Does the one with the subscript j imply that we produce a distribution over the elements of each ui and the one without the j subscript mean a distribution over all the ui vectors?

I thought it could be a typo, but it happened again in another paper Reinforcement Learning for Solving the Vehicle Routing Problem. Here:

enter image description here enter image description here


The notation $a^i_j = \text{softmax}(u^i_j)$ is just an common abuse of notation which should really be written as $a^i_j = \text{softmax}(u^i)_j$, since $u^i$ is the vector which you must compute softmax on, and then you want the $j$th element of the resulting vector. The subscript is then left out of $P(C_i|C_1,\ldots,P) = \text{softmax}(u^i)$ because the LHS is a categorical probability distribution whose parameters are a vector given by the RHS.

I believe the same thing is responsible for the confusion in the second paper, although I haven't read it in detail to be sure.

  • $\begingroup$ Thanks, @shimao. From what you've written, it looks like there is no difference between the equation for $a^i$ (a vector of length $n$) and $P(C_i = j|C_1,\ldots,C_{i-1},\mathcal{P})$ (also of length n). Is this correct? $\endgroup$ Jan 21 '20 at 2:04
  • $\begingroup$ Also, what is the novelty, then, of using pointer networks? It looks like they're just using a pair of RNNs in an encoder-decoder framework with the decoder performing additive cross-attention with the each of the encoder hidden states. Moreover, the decoder vocabulary is constrained to be the same as that of the encoder. Is this accurate? $\endgroup$ Jan 21 '20 at 2:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.