Relationship between optimal action-value function and optimal value function

Question

I would like to clarify on the relationship between the optimal action-value function of an MDP and the optimal value function as I often get confused between them.

Is it possible to express one of the function using the other one?

Answer 1

Deterministic case

If $V(s)$ is the optimal value function and $Q(s,a)$ is the optimal action-value function, then the following relation holds: $$ Q(s,a) = r(s,a) + \gamma V(s') $$ where $r(s,a)$ is the single transition reward, $\gamma$ is the discount factor, and $s'=f(s,a)$ is the next state, given state $s$ and action $a$.

Stochastic case $$ Q(s,a)=r(s,a)+\gamma\sum_{s'}p(s'|s,a)V(s') $$ where $p(s'|s,a)$ is the transition probability to new state $s'$.

For both cases

The relation is the following:

$$ V(s) = \max_a Q(s,a) $$