TF Keras ValueError: Shapes (None, 3, 3) and (None, 3) are incompatible When running my LSTM model, in which I want to take an input (x,y) and output a sequence [(x1,y1), (x2, y2)..., (x,y)] I get a ValueError.
I've read other problems like this, but I still am unable to grasp how I'd fix my model, although I know what the problem is.
My model:
def lstm_functional(X, Y):
    # Input, LSTM and output layers using the functional API
    inputs = keras.Input(shape = (2,1)) 
    # 2,1=X.shape[1],X.shape[2] (timesteps and features)
    lstm = layers.LSTM(64, activation=activation, return_sequences = True)(inputs)
    outputs = layers.LSTM(3, activation=activation)(lstm) # 3=X.shape[0], samples

    model = keras.Model(inputs=inputs, outputs=outputs, name="model_functional")
    model.compile(
        loss="categorical_crossentropy",
        optimizer=keras.optimizers.RMSprop(),
        metrics=["accuracy"]
    )
    model.fit(X, Y, batch_size=2, epochs=2)
    return model

if __name__ == "__main__":

    model = lstm_functional(X, Y)
    

To simplify the problem I created an input (X) and output (Y) which I use to test my model.
My input and output data:
X = np.array([(i, i*2) for i in range(1,4)]) 
Y = np.array([[(i, i*2, i*3)]*i for i in range(1,4)])
print(X)
# [(1, 2), (2, 4), (3, 6)]
print(Y)
# [[(1, 2, 3)], [(2, 4, 6), (2, 4, 6)], [(3, 6, 9), (3, 6, 9), (3, 6, 9)]]

Y = np.array(Y)
X = np.array(X)
X = X.reshape(X.shape + (1,))
print(X.shape)
# (3, 2, 1)

In the data my X values are always a tuple of length 2, but the length of my Y value varies: a list with a varying amount of tuples of length 3.
In the code above I also convert X and Y to a numpy arrays and reshape X, as an LSTM takes in three-dimensional input.
If I try to run my code at this point I'll get an error:
ValueError: Failed to convert a NumPy array to a Tensor (Unsupported object type list).

I assume the reason for this is that the output data (Y) is not of uniform length. Is there a way to use output data of varying length?
What I did was pad my Y sequence to make it uniform:
Y = pad_sequences(Y, maxlen = max(len(i) for i in Y), padding = "post", value = (0,0,0))
print(Y.shape)
# (3, 3, 3)

And now I run the model again, but I get the following error:
ValueError: Shapes (None, 3, 3) and (None, 3) are incompatible

The problem is the final output layer: the output from the output layer (None, 3) does not match with the given Y shape (None, 3,3). None stands for the batch size, which can be altered and is not static, therefore None. How can I reshape/modify/process the Y data so it fits the output of the model?
Just for clarification I'll add the summary of my model:
model.summary()
Model: "model_functional"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_1 (InputLayer)         [(None, 2, 1)]            0         
_________________________________________________________________
lstm (LSTM)                  (None, 2, 64)             16896     
_________________________________________________________________
lstm_1 (LSTM)                (None, 3)                 816       
=================================================================
Total params: 17,712
Trainable params: 17,712
Non-trainable params: 0
_________________________________________________________________
None

I could fix the problem by flattening the individual samples in Y as well as changing the model's output to 9:
Y = Y.reshape((Y.shape[0], Y.shape[1]*Y.shape[0])) # shape (3,9)

But in this case I lose the tuples in the list, which I want to keep, as every tuple corresponds to a step in the sequence.
Questions

*

*Is there a way to use output data (Y) that is not of uniform length?

*How can I fit the Y data to the model, so that the individual tuples in the Y sequence remain, meaning that when i use model.predict(X,Y) I will get a list of tuples where the tuples have a length of 3.

 A: If I understand correctly, you want a model that maps a 2D vector to a (variable-length) sequence of 3D vectors. This is a one-to-many architecture. One way to implement it is to use the decoder part of the seq2seq model. There is a nice blog post on implementing the seq2seq model in Keras.
Since your input is a vector of a fixed size, you can use a dense layer for an encoder, which will produce the state for the LSTM decoder starting from the 2D input. The decoder will then generate a sequence of 3D vectors starting from that state. To indicate the end of the sequence, the decoder can output a special STOP token; I will use a null vector for it. The shape of the output must be the same for all examples, so, indeed, you need to pad it, leaving space for the STOP token. After the padding, Y in your example should look like
targets = [
    [[1, 2, 3], [0, 0, 0], [0, 0, 0], [0, 0, 0]]
    [[2, 4, 6], [2, 4, 6], [0, 0, 0], [0, 0, 0]]
    [[3, 6, 9], [3, 6, 9], [3, 6, 9], [0, 0, 0]]
]

The decoder network also needs inputs. The input at the first time step will be a null vector. During prediction, the output of the decoder obtained at time step $t$ will be fed into it as the input at time step $t + 1$. However, in Keras this cannot be done in training. Instead, one can use the true target at time step $t$. Then the tensor of training inputs for the decoder is the same as the targets above, but shifted by one position along the time axis:
decoder_inputs = [
    [[0, 0, 0], [1, 2, 3], [0, 0, 0], [0, 0, 0]]
    [[0, 0, 0], [2, 4, 6], [2, 4, 6], [0, 0, 0]]
    [[0, 0, 0], [3, 6, 9], [3, 6, 9], [3, 6, 9]]
]

A simplistic model can be constructed as
encoder_inputs = keras.Input(shape=(2,), name='x')
state_h = layers.Dense(latent_dim, activation='relu', name='state_h')(encoder_inputs)
state_c = layers.Dense(latent_dim, activation='relu', name='state_c')(encoder_inputs)

decoder_inputs = keras.Input(shape=(max_len, 3), name='decoder_inputs')
lstm = layers.LSTM(latent_dim, return_sequences=True, name='lstm')
out = lstm(decoder_inputs, initial_state=[state_h, state_c])
out = layers.Dense(3, activation='relu', name='output')(out)

model = keras.Model([encoder_inputs, decoder_inputs], out)
model.compile(loss='mse', optimizer='rmsprop')

(I switched to the MSE loss as the cross-entropy doesn't make sense with this example.) Here, max_len is the length of the output sequence, including the padding and the STOP token, and latent_dim is the number of units in the LSTM decoder. The model's summary looks like this:
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to
==================================================================================================
x (InputLayer)                  [(None, 2)]          0
__________________________________________________________________________________________________
decoder_inputs (InputLayer)     [(None, 4, 3)]       0
__________________________________________________________________________________________________
state_h (Dense)                 (None, 8)            24          x[0][0]
__________________________________________________________________________________________________
state_c (Dense)                 (None, 8)            24          x[0][0]
__________________________________________________________________________________________________
lstm (LSTM)                     (None, 4, 8)         384         decoder_inputs[0][0]
                                                                 state_h[0][0]
                                                                 state_c[0][0]
__________________________________________________________________________________________________
output (Dense)                  (None, 4, 3)         27          lstm[0][0]
==================================================================================================
Total params: 459
Trainable params: 459
Non-trainable params: 0
__________________________________________________________________________________________________

(with max_len = 4 and latent_dim = 8). Train it with
model.fit({'x': x, 'decoder_inputs': decoder_inputs}, targets)

where x is a array of the 2D vectors.
To construct the prediction on new data, you would compute the state of the decoder by applying the dense network. Feed a null vector to the decoder as the first input to get the first 3D output. Then feed this output as the new input, and so on. The generation of the sequence would stop when the decoder outputs a vector close enough to the STOP token.
Let me close with some remarks:

*

*Using the STOP token is probably not the best approach if you want to output $\mathbb R^3$ vectors. Alternatively, you can have another head that predicts the length of the sequence given the 2D input, and then you can sample that number of elements.

*This answer suggests feeding the decoder the same input at all time steps. I guess that's also an option.

