2
You mention two options, $\epsilon=0$ and $\epsilon>0$.
For $\epsilon=0$ there is no exploration, it's all exploitation, so your agent will always select the arm with the maximal reward. If you star from the beginning and all the rewards are random or 0 then this is a very poor choice (you may still get some exploration if you break ties at random).
For $\...
1
After slightly simplifying the notation, by the properties of the exponential function:
$$
\frac{e^{a-b}}{e^{a-b} + e^{b-b}} = \frac{e^a e^{-b}}{e^a e^{-b} + e^b e^{-b}} = \frac{e^a}{e^a + e^b}
$$
Here you can double check.
Using your example,
> exp(2)/(exp(2) + exp(1))
[1] 0.7310586
> exp(2-1)/(exp(2-1) + exp(1-1))
[1] 0.7310586
Notice that you ...
1
First of all, for the continuing case, the return would like something like: $$-\sum^{\infty}_{i=1}\gamma^{K_i-1}$$ where the $K_i$'s are the number of steps until the first $(i=1)$ failure, second failure $(i=2)$ and so on. To help with understanding why it's like that, imagine the following sequence of rewards:
$$0,0,0,-1,0,-1,0,...$$
which is equivalent ...
Only top voted, non community-wiki answers of a minimum length are eligible