0
$\begingroup$

I have a problem where I am mapping from 1D input sequences of length L to 1D output sequences, also of length L. These sequences contain numerical data. The input sequence is the time evolution of a particular physical quantity of a material, and the output sequence is the time evolution of a related physical quantity of the material.

When I try an encoder-only transformer architecture, the performance is rather poor. (This is also true for an encoder-decoder transformer architecture).

However, curiously, if instead of a [batch_size, L,1] tensor, I pass in a [batch_size, 1, L] vector to the encoder-only transformer, i.e. single vectors with dimension L (and outputs are also [1,L]), the performance is much better.

I have two questions:

  1. My understanding was that the true power of the attention mechanisms is realised when operating on sequences of vectors, allowing the model to dynamically weigh the importance of different sequence elements relative to each other. Why might this not be the case here? Especially since the input and output sequences are time series?
  2. What even does the attention mechanism do if there is just a single vector input to the transformer?
Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ The poster omitted the batch dimension in the description but apparently took care of it. Could you please update your post. After all he seems to ask how it can be that the transformer performs differently, depending on how he submits the data. It is clear that the temporal dimension is treated differently when the attention matrix is computed. If is not clear how this leads to a degradation here. This seems to be the question. $\endgroup$
    – Ggjj11
    Commented Mar 24 at 19:04
  • $\begingroup$ @Ggjj11 I have updated the question to incl. batch size. And that's a good summary of the question :) $\endgroup$
    – Blahblahblacksheep
    Commented Mar 24 at 20:42

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.