Sutton et al. use throughout their book Reinforcement Learning capital letters to describe random variables. At page 131 they introduce Q-Learning.
$Q(S_t,A_t)\leftarrow Q(S_t,A_t) + \alpha [R_{t+1} + \gamma \max_a(S_{t+1},a) - Q(S_t,A_t)$
Why $S_t, A_t$ are capital letters and thus random variables? $S_t$ is the actual state where we are at the moment and $A_t$ is the action we executed. That said we have some concrete realisations $S_t=s$ and $A_t=a$.