0
$\begingroup$

According to the PyTorch manual, if you want to feed a batch of variable-length sequences to a sequence model, you pad and pack the batch before feeding it to the sequence model. That is,

lstm = LSTM(bidrectional=True)

packed_batch = pack_padded_sequence(batch)
output, (h_n, c_n) = lstm(packed_batch)

In this case, if I do output[-1, :, :], what would be the result?

Suppose that the first sequence in the batch has the length 7 and that the maximum length is 10 -- that is, the first sequence will receive 3 paddings.

If I do output[-1,0,:] (to retrieve the last hidden state of the first sequence), does it return the concatenation of the 7-th hidden states of the bi-LSTM (which is the correct)? Or, would it return be still the concatenation of the 10-th hidden states?

$\endgroup$

put on hold as off-topic by kjetil b halvorsen, mkt, Peter Flom 7 hours ago

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This question appears to be off-topic because EITHER it is not about statistics, machine learning, data analysis, data mining, or data visualization, OR it focuses on programming, debugging, or performing routine operations within a statistical computing platform. If the latter, you could try the support links we maintain." – kjetil b halvorsen, mkt, Peter Flom
If this question can be reworded to fit the rules in the help center, please edit the question.

0
$\begingroup$

output[-1,:,:] will be the last slice of the tensor, i.e., the padding.

The real final states are in the second member of the tuple returned by lstm.

If you really want to get the state from output, you can use torch.gather.

torch.gather(output, dim=0, index=lengths - 1)

where lengths is a 1-D tensor with lengths of the sequences in the batch.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.