I am working on training a reinforcement learning agent with a huge dataset of past human players' experience. Each user had to independently increase their game score. I am using the dataset because I have plenty of data and because on-policy training can be slow due to the game speed.

The only problem I have is the human player ID as a categorical feature. I have around 1000 experience play from each user and an ID should be assigned so the LSTMs can detect the sequential behavior of the game play.

Should I:

1- Serve each player's experience as an episode to the agent so to get rid of user IDs as a category

2- or keep the training batches together including all the user IDs and train the agenet with them?

The problem here is that it is complicated for me to use one-hot encode for this large category.


If you're not explicitly modelling the player's behavior, then using IDs doesn't make any sense. The point of using categorical features is to use them in prediction - the player IDs aren't part of environment, right?

  • $\begingroup$ The player IDs were proposed so the agent during the pre-training understands which user started which game i.e. the training batch contains several users with their current state/action at a certain date. Now, if we serve the training batches player by player then there's no need for their IDs. $\endgroup$ – Leb_Broth Apr 7 '18 at 15:05
  • $\begingroup$ How are you going to use these IDs for unseen data? $\endgroup$ – Jakub Bartczuk Apr 7 '18 at 15:35
  • $\begingroup$ The data is the state/action for each player in the past. Assigning to them their IDs in this case, would let the LSTM get the temporal dependency for each player. Am i missing something? $\endgroup$ – Leb_Broth Apr 7 '18 at 15:39
  • $\begingroup$ What I mean is why do you even use this data? It seems to me that RL agent is supposed to react to environment, and I don't think what player ID has to do with that $\endgroup$ – Jakub Bartczuk Apr 7 '18 at 15:42
  • $\begingroup$ Yes exactly. But my environment is very slow for the agent to learn from it. That is why i decided to pre-train it by demonstration. Is there an alternative for the training in case of slow environment? Should I create my own gym maybe in an accelerated way? $\endgroup$ – Leb_Broth Apr 7 '18 at 15:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.