# q learning - how to find optimal policy after following random policies?

Editing the original question.

I am trying to use Q learning to figure out an optimal policy to follow given initial conditions. My question is regarding what to do after learning Q values.

I have a history of different episodes where in each episode, I have followed random actions in different states until I reach final state. I use Q learning algorithm as mentioned on wikipedia and other sources to update the Q values.

As I understand, now that I have the Q values, the optimal policy to follow is to select the action with highest Q value in a given state- i.e. observe the initial state s, take action with max(Q(s,a), move to second state s' and take action with max Q(s',a) ....and so on until you reach final state.

My question- How do I figure out the optimal policy when I can't keep track of the states. Given initial state s, is there a way to just output a sequence of actions to take without keeping track of what states result from each individual actions?

Maybe it can become more clear that all of above works if you think about Q-Learning with function approximation. Q(s,a) is an estimate of the discounted future return given current state and action. You're essentially solving a regression problem: at each step features are $(s_t, a_t)$ pairs (converted to vectors), and target variable is $r_{t+1} + \gamma Q(s_{t+1}, a_{t+1})$. Not using $s_t$ or using fixed $s_t$ means that different features are used, nothing more; you're free to change features as you want.