What happens when number of arms in a bandit problem increases? I'm doing simulations of multi armed bandits in various settings. 
My question is what happens when the number of arms increases. My settings are 1000 trials and 500 experiments. 
But there strange behaviour that I can't understand when I increase the number of arms from 10 to 1000. 
10 arms case plot:

1000 arms case plot:

Why greedy method performs much better than UCB and softmax? 
 A: As I can see here, you have kept the no. of iteration as 1000, across both 10 arms case and 1000 arms case. 
Consider UCB for example, it pulls all the arms at least once before it starts building confidence bounds completely. For the 1000 arms case, the number of iterations is enough to just pull all the arms once and no more moves left. This is obviously bad when compared to epsilon-greedy or softmax, where the "optimal" action is pulled with a higher probability. 
So you will have to increase the number of iterations (should be at least 10 times greater than no. of arms) for UCB to make sense.
In the case of epsilon-greedy and softmax (for 1000 arms), from your plots, epsilon-greedy seems to perform better than softmax but both of them are worse (percentage of optimal action is less than ~5%). 
My intuition is that softmax should perform better than epsilon-greedy in this case because, even though both of them do a bit of exploration, epsilon-greedy's exploration is very small (epsilon/1000 for non-greediness) when compared to softmax. You need to adjust the temperature parameter in softmax to obtain optimal results (which could be better than epsilon-greedy according to my intuition).
