q learning - how to find optimal policy after following random policies?

Question

Editing the original question.

I am trying to use Q learning to figure out an optimal policy to follow given initial conditions. My question is regarding what to do after learning Q values.

I have a history of different episodes where in each episode, I have followed random actions in different states until I reach final state. I use Q learning algorithm as mentioned on wikipedia and other sources to update the Q values.

As I understand, now that I have the Q values, the optimal policy to follow is to select the action with highest Q value in a given state- i.e. observe the initial state s, take action with max(Q(s,a), move to second state s' and take action with max Q(s',a) ....and so on until you reach final state.

My question- How do I figure out the optimal policy when I can't keep track of the states. Given initial state s, is there a way to just output a sequence of actions to take without keeping track of what states result from each individual actions?

Answer 1

I'm not sure I understood the question correctly, but if you don't have state information available then nothing prevents you to use just Q(a) instead of Q(s, a), i.e. assume that Q(s,a) doesn't depend on s. Q-learning still works in this case.

Alternatively, if you have initial conditions available, you can use it as a state, keeping s the same for all steps of a trajectory, but varying it between trajectories.

Maybe it can become more clear that all of above works if you think about Q-Learning with function approximation. Q(s,a) is an estimate of the discounted future return given current state and action. You're essentially solving a regression problem: at each step features are $(s_t, a_t)$ pairs (converted to vectors), and target variable is $r_{t+1} + \gamma Q(s_{t+1}, a_{t+1})$. Not using $s_t$ or using fixed $s_t$ means that different features are used, nothing more; you're free to change features as you want.