0
$\begingroup$

According to the PyTorch manual, if you want to feed a batch of variable-length sequences to a sequence model, you pad and pack the batch before feeding it to the sequence model. That is,

lstm = LSTM(bidrectional=True)

packed_batch = pack_padded_sequence(batch)
output, (h_n, c_n) = lstm(packed_batch)

In this case, if I do output[-1, :, :], what would be the result?

Suppose that the first sequence in the batch has the length 7 and that the maximum length is 10 -- that is, the first sequence will receive 3 paddings.

If I do output[-1,0,:] (to retrieve the last hidden state of the first sequence), does it return the concatenation of the 7-th hidden states of the bi-LSTM (which is the correct)? Or, would it return be still the concatenation of the 10-th hidden states?

$\endgroup$
0
$\begingroup$

output[-1,:,:] will be the last slice of the tensor, i.e., the padding.

The real final states are in the second member of the tuple returned by lstm.

If you really want to get the state from output, you can use torch.gather.

torch.gather(output, dim=0, index=lengths - 1)

where lengths is a 1-D tensor with lengths of the sequences in the batch.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.