0
$\begingroup$

I have created a model for text generation using LSTM. I am having chess sequences learned, reporting only the pieces moved during the moves. So when I move a pawn on my game there will be "p", a bishop "b" and so on. Obviously this data has been mapped and taken in integer. The model used for learning is:

model = Sequential()
model.add(Embedding(vocab_size, 5, input_length=seq_len))
model.add(LSTM(256, return_sequences=True))
model.add(LSTM(256))
model.add(Dense(256, activation='relu'))
model.add(Dense(vocab_size, activation='softmax'))

My model at the end of 100 epochs has an accuracy of 0.55 and a loss of 1.05. I cannot increase these metrics. I have seen a few articles that have the same performance as mine, so I was asking if it is normal to have these values at the end of the train for models with LSTM. If they are acceptable, how can I do an evaluation for models that use LSTM? What is the best way to do the evaluation?

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ Some things are easier to predict than others, so it's not really fair to compare your model for sequences of chess moves with a model of temperature or natural language (for example). // What do you mean by "What is the best way to do the evaluation?" You already have the accuracy and the loss values. What are you evaluating? What additional information do you want to learn? $\endgroup$
    – Sycorax ♦
    May 9 at 13:46
  • $\begingroup$ Thanks for your reply @Sycorax . I have the acuracy and loss of the model during training, but I have no idea how to do some sort of evaluation on them. $\endgroup$
    – ValerioGoretti
    May 9 at 14:13
  • $\begingroup$ Apply the model to some hold-out partition and measure the statistics of interest (e.g. accuracy or some other metric). It's not clear if you expect this is different than any other neural network or machine learning model, but so far you haven't articulated a reason that it would be. $\endgroup$
    – Sycorax ♦
    May 9 at 14:21
  • $\begingroup$ Ok thanks @Sycorax , now I'll try this method and try to understand what is the better metrics to analyse. My difficulty is in understanding how to evaluate a text generation model. Since the text generated during prediction is new text, I don't know how to evaluate it. $\endgroup$
    – ValerioGoretti
    May 9 at 17:27

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.