The question is basically
- What should I do if state vector has variable length?
- If the action is bounded and continuous, how can I obtain max(Q(state,action)) without using painfully slow global maximization?
So I was studying reinforcement learning in Space-Invader-like games.
The game itself actually is quite complex, with 5 types of invaders (straight, circle, chasing, random-walk and almost stationary mothership) and lots of randomness inside. Player can choose its velocity(0-1) and direction. A special MP bar is used to generate AOE attack, with MP regen ~ (1-vel)^2
Because the system is way too complex for Q-learning to learn with reasonable speed, I applied a preprocessing ANN first, to identify invaders, lasers and player status (HP, MP, scores, etc).
So I got a list of identified invaders with features like position, velocity, predicted velocity, HP, ticks from fire laser.
However the features now is no longer a fixed-length vector. Also the action is continuous.
I have some ideas but not sure if they're efficient or not.
- One neural network per input size. So maybe I have to create dozens of ANNs to cover the entire state-space. (however I'm sure this is not the best way. How can 39 invaders be any different from 40 invaders...)
- Discretize the action into, say, 314 points (so 314 output neurons) and use interpolation to get Q-function (but the massive size of network surely will kill the performance)