I am implementing Q-Learning to find the "Next Best Action" a sales rep should take on a particular account to collect more money.
I am feeding the algorithm past actions taken on the accounts and the resulting states from those actions in order to learn the Q matrix. My problem is not every account had all available actions taken on it.
So in the first iteration of the Q-Learning algorithm we randomly select an initial state s
(In my application this would be a unique account z
AND state s
combination). We then randomly chose an action a
to execute. Unfortunately we may not have a ever taken action a
on that particular account z
, and as a result cannot observe any resulting state s'
. Note: State transitions are constrained to one account. For example: I can't invoke an action a
on account z1
and observe the resulting state on a different account z2
.
If this scenario occurs should I just end the episode? However I never reached the goal state. Do I violate any assumptions for ending the episode early for reasons other than reaching the goal state?
Is there a better way to implement Q-Learning using past data that does not include records for all state and action combinations?
Should I even be using Reinforcement Learning, or should I use Markov Decision Processes since I can compute the transition matrix using my past records of the system?