Q-learners with discretised Q-values I have a Q-learning algorithm with a finite number of states. Each state, however, is represented by the usual 32 bit Q-value. These Q-values allow to carry information about the previous experience far beyond the state of the Q-learner (in my case the state of the Q-learner is the sequence of his own choices and rewards). 
What if the Q-value could be only binary? Then Q-learner wouldn't be able to carry that information and wouldn't learn much. I am very interested in how much memory, in bits, does the Q-learner need to learn different rules. (For example. Q-learner always has two choices, 1 and 0. If he chooses 1 1 0 in a row he gets the reward.)
However, I have problems with implementation. Q values are bounded only by experience, so how exactly do I implement the update rule for Q-value if I want to constrain that Q-value can be only one of n different values? 
My update rule
Then $Q_{new} = Q_{old} + \alpha (R + \gamma  * \max(Q_{future\ choice\ 1} - Q_{future\ choice\ 0}))$
How do I now map this new value on the limited set that Q-value can be?
If $\alpha = 1 , \gamma = 1; R,Q $ are elements of $\{0,1\}$ then $Q_{new}$ is element of $\{-1, 0, 1, 2\}$ And mapping is easy. But what if it's different?
Has anybody studied that? Papers and any suggestions are appreciated.
P.S. I am not sure that this is the best place for machine learning questions. If you know another Stack Exchange site that deals with that or any other site, please let me know.
 A: If you know the minimum and maximum values for the reward (let us denote them $R_{min}$ and $R_{max}$), you can easily derive the minimum and maximum values for the $Q$-value, by a simple application of power series: 
$$
Q_{max} = R_{max} + \gamma R_{max} + \gamma^2 R_{max} + \ldots = \frac{R_{max}}{1-\gamma}
$$  
Similarly, $Q_{min} = R_{min} + \gamma R_{min} + \gamma^2 R_{min} + \ldots = \frac{R_{min}}{1-\gamma}$.
As an example, in the case where $R_{max} = 5, R_{min}=-5 \text{ and } \gamma = 0.9$, we know that the $Q$-values will be lying in the interval $[-50, 50]$.
Given these bounds, you can easily divide the interval into $n$ buckets of equal size, where $n$ is the number of distinct discrete values you want to use for the $Q$-value. The boundaries for these intervals will be 
$$
Q_{min} + i \frac{Q_{max}-Q_{min}}{n} \text{ for } i = 1, 2 \ldots (n-1)
$$
When doing the Q-learning update, you can then simply check in which of the $n$ buckets the new Q value will fall into, and map your initial, real-valued Q estimate to this discretised value. To make things easier, the discretised values for each bucket might correspond to the mean value for the sub-interval.  In the example above, one could use the two values -25 and 25 as representative of the two intervals $[-50,0]$ and $[0,50]$.
