Why can constant alpha be used for Q-Learning in practice? The convergence criteria of Q-Learning state that the learning rate parameter  $\alpha$ must satisfy the conditions:
$$\sum_k \alpha_{n^k(s,a)} =\infty \quad \text{and}\quad \sum_k \alpha_{n^k(s,a)}^{2} <\infty \quad \forall s \in \mathcal{S}$$
where $n_k(s,a)$ denotes the $k^\text{th}$ time $(s,a)$ is visited
Why can a constant $\alpha$ be used in practice?
 A: The same question can be asked in principle for any machine learning method. We almost never decay the step-size, and in fact we often use optimizers (like ADAM) that have errors in their proof of convergence. In fact, we continue to use ADAM instead of alternatives that correct this deficiency and provably converge (see here).
Why the discrepancy? Perhaps convergence is not something that matters too much in practice. It's still important to be sure, but I would rather have something that is quickly able to reach a small region of the optimum and never converge than something that guarantees convergence in an unachievable limit. I think this is the key point as well. An algorithm that converges with a decaying step-size will still get close to a local optimum with a constant step-size. However, the constant step-size will determine how close one can get before oscillating around said optimum.
The problem with such a set-up is that it is hard to analyze. What can be said about the finite-time analysis of algorithms in the average case? Not much. A convergence proof is easy to show in comparison. 
You can also read a similar discussion in Chapter 2, page 33 of Sutton's RL book. Although the chapter is on bandits and not on Q-learning, Sutton discusses that constant step-sizes can be good in non-stationary problems, where convergence is not really desirable. 
