1
$\begingroup$

I have been making tests with Transformer model provided on Keras.io page, training for classification and seq2seq tasks in several datasets and compare Transformer to GRU/LSTM with almost same number of parameters in both models.

In all my experiments, everything but the model are controlled variable and they are constant. I conducted my tests with GPU on my desktop (GTX 1070), Google Colab GPU and AWS Tesla T4.

What I witnessed are:

  • Training Transformer model takes longer for same number of epochs
  • Both training and validation loss are much higher for Transformer model, than GRU/LSTM model at the end of same number of epochs.

I tried various number of "number_of_heads" and "number_of_transformer_blocks", result did not change really.

Transformers were said to be superior to RNNs like GRU/LSTM but I have never witnessed that. Therefore my question is, am I missing something?

Improve this question
$\endgroup$
6
  • $\begingroup$ Are you tuning the hyperparameters of the models? Which ones? How? $\endgroup$
    – Sycorax ♦
    Commented Sep 10, 2021 at 13:51
  • $\begingroup$ I am not tuning hyperparameters, I use same parameters whenever possible such as optimizer (adam), learning rate, embedding size. Such a huge gap for vanilla models doesn't sound right. I can provide a comparison Notebook on Colab soon. $\endgroup$
    – meliksahturker
    Commented Sep 10, 2021 at 13:56
  • 2
    $\begingroup$ Tune the hyperparameters, especially the learning rate. The question you're implicitly trying to answer is "how does the best transformer network compare to the best LSTM/GRU network?" and the only way to get the best version of a network is to tune it. $\endgroup$
    – Sycorax ♦
    Commented Sep 10, 2021 at 13:58
  • 1
    $\begingroup$ You're saying that one network being better than an alternative, comparably-sized network is not a demonstration that one is "superior" to the other? What does "superior" mean to you? // Anyway, there's plenty of material about tuning neural networks that you can find using a search. Here's one example to get you started. stats.stackexchange.com/questions/342462/… $\endgroup$
    – Sycorax ♦
    Commented Sep 10, 2021 at 14:07
  • 1
    $\begingroup$ I am facing the same here, I also read about it in a scientific paper although the dataset is medium size. I notice that (the RMSprop optimizer) outperforms Adam, at least in my case for time series classification. Also, I noticed that they need more epochs to reach a good result compared to other models. I read about using a cyclical learning rate, but I did not try it yet . $\endgroup$
    – Malak_MAM
    Commented Feb 4, 2023 at 21:08

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.