There is a sizable body of work under the title "contextual bandits", which might be of interest to you.
A bit more information. Contextual bandit algorithms attempt to solve the problem of picking the arm of a multi-armed bandit conditioned on some random vector (called the context). There is a discussion of the problem in this blog post. The traditional application is for internet advertising, where say the system sees a user's query (which is the context) and has to pick from one of K possible ads to show.