I have simple implementation of Q-learning algorithm and I'm trying to run it on
States space size = 36865 Actions space size = 25
So my resulting Q-table is basically 1 million items table.
- Is there a definition for problem sizes (small/medium/large) based on action/space size and corresponding algorithms which fits best for those
Using implementation below I'm wondering
s_S = 36865 s_A = 25 alpha = 0.1 # learning rate gamma = 0.9 # discount factor eps = 0.4 # exploration factor Q = np.zeros(shape=(s_S, s_A)) EPOCHS = 10 for i in range(EPOCHS): # reset env for each epoch agent = Agent(env=env) s = 0 # starting state while s is not None: # rollout a = get_next_action(agent, Q, s, eps) r, s_ = agent.take_action(a, s) max_q = maximize_q(agent, Q, s_) # maximize Q for the next state Q[s, a] = alpha*(r + gamma*max_q - Q[s, a]) s = s_ policy = extract_policy(agent, Q) # evaluate agent behaviour under the policy
- Can I run N threads in parallel and share
Qtable between them? Will this approach converge or will I end up in a state with corrupted
Qtable? Is there any other way to speed up process for such a huge table?