I have used keras.preprocessing text tokenizer to fit on the training data alone, computed the (train) vocab size 'input_dim' and maximum train sequence length 'input_length' before fitting my neural network model with an embedding layer(input_dim, output_dim, input_length).

When I try to process the test data and predict my trained model, I am not sure about the effect of applying the tokenizer fitted on the train data to my test data. In particular I did not re-compute the input_dim and input_length for test data, and used trainTokenizer.texts_to_sequences() on the test data straightaway. Processing it with the train tokenizer ensures the same input_dim and input_length, such that the test data has a compatible shape when passed to predict function.

However the original test data has greater max sequence length and I wonder if there is information loss in what I have done.
But otherwise I do not think it is statistically appropriate to concatenate the train and test data to fit tokenizer and train model. How should I process my test data such that the dimensions are appropriate for feeding to the model, and simultaneously keep test data 'unseen' in developing the model?

Any clarification is welcomed!


1 Answer 1


The point of using a test set is to validate your model on unseen data. To do this, you need to apply exactly the same preprocessing and prediction function to the test set.

If you used different preprocessing and observed differences between train and test metrics, you wouldn't know if the the culprit is preprocessing or the model itself. With different preprocessing, in most cases, you would need to re-train the model on the test set for the new data format, so you could as well don't use the test set, just train on all data and keep your fingers crossed it would work on out-of-sample data.

If there are drastic differences you should rather ask yourself why is it so. Was the split of the data done at random? Which train or test data is the “correct” representation of the real-life data? Maybe you need to collect more, or better, data?

  • $\begingroup$ To confirm, with "exactly the same preprocessing", in this case each of my test sequence will have to be the same length as my train data (because the model is developed based on the train input_length)? $\endgroup$
    – siegfried
    Sep 17 at 5:39
  • $\begingroup$ @siegfried yes. But there are also models for varying-length inputs like RNNs, where it wouldn't matter. $\endgroup$
    – Tim
    Sep 17 at 5:42
  • $\begingroup$ right rnn layer itself does not care about the sequence length. but when the rnn model has an embedding layer where an argument is input_length (the max sequence length), i guess it will have to be the max sequence length from the train data. test data will have to compromise on max sequence length when predicting? $\endgroup$
    – siegfried
    Sep 17 at 5:50
  • $\begingroup$ @siegfried yes. $\endgroup$
    – Tim
    Sep 17 at 6:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.