You're right, I didn't actually write a sentence which was an explicit question... If this were an analogous question but describing the vanilla MAB setup, I'd be very pleased with answers which either pointed to or described e.g. UCB, Thompson sampling, along with notes on assumptions required, regret analysis, whatnot. It seems like literature on the problem I'm describing is either absent or rare, though, so the question is basically 'do you know any literature, or have an insight into heuristics or solutions for this problem?'

Thanks! Yes. I would certainly be interested in any solutions (or pointers or gestures) which leaned on further assumptions, for example boundedness or unimodality or subgaussian-ness or even Gaussian

@AleksejsFomins it sounds like you indeed grasp the problem. I agree, once an island has yielded something of value, an important consideration for subsequent islands is the estimated spread of their yields, as well as any central tendency. So a policy should take that into account. But, like in the standard MAB problem, there remains an issue of efficient exploration.

I bet someone smarter than me could give this a full Bayesian treatment

If I have an index-based approach to solving an otherwise-similar $k=1$ MAB problem, how much do I lose by simply selecting the top-$k$-indexed values for this modified problem? If I had a Thompson-style approach to an otherwise-similar $k=1$ MAB, how much do I lose by simply sampling my set-of-$k$ from the posterior distribution over top-$k$-means?

I wanted to add a tag exploration-exploitation which I thought would be appropriate, but don't have the reputation required on this site. Perhaps multiarmed-bandit is sufficient

I'm more than happy with answers assuming fixed $n$ and $k$ per round, but of course welcome any more general answers as well!