All value-based reinforcement learning (RL) methods use some form of backup between time steps. The backups take values known from later time steps, and use it to calculate values expected at earlier time steps.
This is an important part of RL method theory and design, and there is a visualisation of it called "backup diagrams" which you will find many examples of in the early chapters of Reinforcement Learning: An Introduction.
In Q learning, you can see the relationship between values at $t$ and $t+1$ in the update:
$$Q(s_t, a_t) \leftarrow Q(s_t, a_t) + \alpha(r_{t+1} + \gamma \text{max}_{a'} Q(s_{t+1}, a') - Q(s_t, a_t))$$
That is, the value of $Q(s_t, a_t)$ is being updated, and the values of $r_{t+1}$ and $s_{t+1}$ are used directly in the caclulation of that update. Over many repetitions of this update rule, values are backed up from later time steps (where you have better knowledge of the eventual rewards seen) to earlier time steps.
Specifically for tabular Q learning, you were concerned about initial inaccuracy of the bootstrap estimates. The resulting bias is a concern, and in some off-policy methods it can prevent covergence. However, with reasonable assumptions in tabular Q learning, it can be proved to converge.
