# What is the difference between “inverse reinforcement learning” and supervised learning?

It will be great if you provide an example too.

• This is rather broad, & may be hard for people to answer. Can you provide some context, perhaps where you've seen these terms used together? – gung - Reinstate Monica Oct 19 '16 at 19:23
• @gung the problem these terms are not used together and i see them a little similar. i want to know when to use 1 method and not the other one – floyd Oct 19 '16 at 20:15

Disclaimer: I am MSc student of Control theory (with engineering background) who is starting his thesis on Reinforcement Learning. I am just beginning to get a feel for the field. Kinda like I just am taking my first walk around the lake of machine learning. So my information may not be spot on. I am answering because I FEEL that I understand the subtle difference. I also get the feeling from your request of example that you would like an application oriented example, not a mathematical abstraction of it.

Differences - IRL frames its problem as an MDP and uses the notion of an 'agent' to select 'actions' that maximize the net reward. The key difference is, in IRL supervised learning techniques (ie data fitting) are used to obtain the reward function. Supervised learning uses labeled data in order approximate a mapping.

Example of learning ground distance from images
Supervised learning: Using features in images with labeled ground distances to train a Neural network weights to find ground distances in the general case.
IRL: Using labeled data to derive a reward function, which would be a mapping from features to rewards. Letting an agent explore the space of features and coming up with a policy that selects the best actions, which in this case would be an estimation of the ground distance.

For this specific task I described, it seems trivial since using RL for the classification of image distance when simpler supervised learning suffices is redundant. However, in RL situations where the definition of reward functions are difficult but it can be advantageous to use RL, IRL can prove very useful.
For example, if one were to imagine using RL to teach acrobatic maneuvers to helicopters (Paper by Abbeel Et Al), using IRL to obtain reward functions can be very useful. Once the reward functions for the maneuvers are obtained, this can be used to teach these maneuvers to others helicopters (with different aerodynamic models but similar controls) how to perform these maneuvers. Using supervised learning to come up with a mapping of states to controls wont work, since the different aircrafts would have different aerodynamic models.

Reference:
* Ng, A. Y., & Russell, S. J. (2000, June). Algorithms for inverse reinforcement learning. In Icml (pp. 663-670).
* Abbeel, P., Coates, A., & Ng, A. Y. (2010). Autonomous helicopter aerobatics through apprenticeship learning. The International Journal of Robotics Research.

Inverse reinforcement learning (IRL) can be seen an instance of supervised learning. The data is the demonstrations and the target is the reward function. So the 'learning' task is just to look for the mapping from the space of demonstrations to reward functions, under the constraints of the specification of the MDP.

Concrete example: Lets use Bayesian IRL to illustrate. Given some MDP without reward function $(\mathcal{S}, \mathcal{A}, T, \gamma)$, and a set of demonstrations $\Xi = (\xi_1, \ldots, \xi_M)$ where each demo trajectory $\xi_i = ((s^i_1,a^i_1), \ldots, (s^i_H,a^i_H))$ is a set of state-action pairs. The BIRL task is to find, $$\Pr(R \mid \Xi)$$ which is easily expanded as $\Pr(R \mid \Xi) \propto \Pr(\Xi \mid R) \Pr(R)$ by Bayes rule. The 'data' ($\Xi$) are also often assumed to be iid. From this formulation it is obvious that its a supervised learning problem. The devil is only in the details of computing the likelihood.

Important: IRL seeks the reward functions that 'explains' the demonstrations. Do not confuse this with Apprenticeship learning (AL) where the primary interest is a policy which can generate the seen demonstrations (although this is often but not necessarily obtained via the reward).

Additionally, there is behavior cloning which is also closely related. Given some examples of a behavior, behavior cloning simple try to reproduce it. This could mean generating behavior that 'matches' the statistics of the observed behavior. It is obvious to see how this is supervised learning. E.g. given some demonstrations, train a neural net to generate 'similar' behaviors given 'similar' situations.

P.S. Forgive my hand-wavy nature with the vocabulary.