What are the available options if I want to perform a scoring task on a set of observations that: a) have a set of variables connected to them and, b) each round I get new information about the success of the latest round.

What I mean is, I start with a normal scoring task: Using the available training data I want to rank each observation in a new dataset. This part is straightforward and can naturally be performed by a range of different methods. However, the nature of problem is that I get feedback from this ranking when see how observations react. Specifically, when a number of observations are picked from those with the highest score, the feedback consists of these observation either reacting positively or negatively to them being picked. This information can in turn be used for the next round, which is performing a new scoring and picking the next observations with the highest scores.

What mathematical methods can be used to process this feedback information? How should/could the feedback be used in conjunction with the original (background) information? What R packages are there for these types of on-line machine learning problems?

  • 1
    $\begingroup$ I am not sure that reinforcement learning is appropriate here. Question for clarification: Do you can get feedback for an observation only once OR can you elect it for feedback multiple times ? $\endgroup$
    – steffen
    Commented Sep 12, 2011 at 16:53
  • $\begingroup$ I get feedback from the chosen observations once for each round. If the observation is not scored to be among the n:th highest in a round, I get no feedback at all for those observation in that round. A observation can be chosen again (and give feedback multiple rounds) if it's scored high multiple times. $\endgroup$
    – Figaro
    Commented Sep 13, 2011 at 6:29

2 Answers 2


It sounds like you want reinforcement learning. I'm having a little trouble parsing the exact details of your specific problem, but perhaps it could be cast in the framework of a Multi-Armed Bandit problem?


alto is right, it sounds like a reinforcement learning setting. Although most RL examples involve a robot, RL can be applied to various other interactive problem domains.

You should read this free ebook by Sutton and Barto, which is the standard work for RL.

There have been loads of developments since the original writing of the book though. I recommend that once you can phrase your problem in RL formalities, you come back and ask what kind of algorithms can be used for that.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.