I am implementing Q-Learning to find the "Next Best Action" a sales rep should take on a particular account to collect more money.
I am feeding the algorithm past actions taken on the accounts and the resulting states from those actions in order to learn the Q matrix. My problem is not every account had all available actions taken on it.
So in the first iteration of the Q-Learning algorithm we randomly select an initial state
s (In my application this would be a unique account
z AND state
s combination). We then randomly chose an action
a to execute. Unfortunately we may not have a ever taken action
a on that particular account
z, and as a result cannot observe any resulting state
s'. Note: State transitions are constrained to one account. For example: I can't invoke an action
a on account
z1 and observe the resulting state on a different account
If this scenario occurs should I just end the episode? However I never reached the goal state. Do I violate any assumptions for ending the episode early for reasons other than reaching the goal state?
Is there a better way to implement Q-Learning using past data that does not include records for all state and action combinations?
Should I even be using Reinforcement Learning, or should I use Markov Decision Processes since I can compute the transition matrix using my past records of the system?