In order to learn an environment, we can use Q-learning as a model free algorithm. In Q-learning, there is a Q function and we update this function using Q-learning algorithm until convergence. When converged, we can find optimal value function for each particular state by maximizing the Q function at that state over all possible actions.
Now, my question is that, assume we don't care about what the optimal action is at each state and we just want to find the optimal value function at each state without calculating the Q function first. Is there anyway for doing this?
By optimal value function, I mean value function corresponding to the optimal policy. Further, we don't have the full model of the environment (rewards and transition probability). So, it is a learning problem.
The reason I am looking for the optimal value function and not optimal Q-function is that, assume the action space is very large but the state space is not. And I only care about the optimal value function. Since the action space is very large, finding optimal Q functions is not efficient or probably impossible for this case.