I'm currently going through this paper:
Bidirectional Attention Flow for Machine Comprehension, Seo, Minjoon, et al. (2016)

They perform a $max_{col}$ function over a matrix $S \in \mathbb R^{TxJ}$:

We obtain the attention weights on the context words by $b = softmax(max_{col}(S)) ∈ \mathbb R^{T}$ , where the maximum function ($max_{col}$) is performed across the column.

Intuitively I would understand "where maximum function is performed across the column" as I taking the maximum from each of the $J$ columns and getting as a result a vector from $\mathbb R^{J}$ (one value for each column), not $\mathbb R^{T}$.

Even though it makes sense having a vector from $\mathbb R^{T}$ in the further course of the paper, the quoted passage really is counterintuitive for me.
(Naively I would rather call it $max_{row}$ what they are doing, am I wrong?)

To make things more confusing when looking after an explanation I found this paper with a similar procedure, but just the opposite outcome:
A Question-Focused Multi-Factor Attention Network for Question Answering, Kundu, S., & Ng, H. T. (2018)

There they have a matrix $A \in \mathbb R^{TxU}$:

We apply a $maxcol$ operation on $A$ which forms a row vector whose elements are the maximum of the corresponding columns of $A$. We define $k \in \mathbb R^{U}$ as the normalized max-attentional weights:

$$k = softmax(maxcol(A))$$

So my questions are:

  • Am I misunderstanding "$max_{col}$ performed across the column"? What is normally meant by that statement?
  • Is there some mistake in one of the papers? To me it seems that both papers using just the same function and getting converse results.

Thanks in advance!

  • $\begingroup$ Both of your links are to arXiv pre-prints, and pre-prints are known to have typographical errors and other mistakes. Have you looked at the print versions of the papers? If no print publication exists (perhaps because it's in press), you could e-mail the authors. $\endgroup$ – Sycorax says Reinstate Monica Mar 19 at 14:55
  • $\begingroup$ @Sycorax Thanks for this suggestion, I will check it! $\endgroup$ – blue-phoenix Mar 19 at 16:44
  • 1
    $\begingroup$ If you what you're looking for, you can use the Answer box below to post what you learn so that other people who encounter this problem will know what accounts for the discrepancy. $\endgroup$ – Sycorax says Reinstate Monica Mar 19 at 21:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.