Return to Question

I find two different types of formulations for Contextual Bandits:

Definition 1: In a contextual bandits problem, there is a distribution $P$ over $(x,r_1,...,r_k)$, where $x$ is the context, $a \in [k]$ is one of the $k$ arms to be pulled, and $r_a$ is the reward for arm $a$. The problem is a repeated game: on each round, a sample $(x, r_1, ..., r_k)$ is drawn from $P$, the context $x$ is announced, and for precisely one arm $a$ chosen by the player, its reward $r_a$ is revealed.

Definition 2: The algorithm observes the current user $u_t$ and a set $A_t$ of arms or actions together with their feature vectors $x_{t,a}$ for $a \in A_t$. The vector $x_{t,a}$ summarizes information of both the user ut and arm $a$, and will be referred to as the context. Based on observed payoffs in previous trials, A chooses an arm $a_t ∈ A_t$, and receives payoff $r_t,a_t$ whose expectation depends on both the user $u_t$ and the arm $a_t$.

The fact that when stating def. 2, the authors of the paper cite the paper from def. 1: Following previous work (def. 1), we call it a contextual bandit. This is very confusing to me.

In particular, def. 1 assumes that only one context is revealed to the learner. In the second formulation, you observe "contexts" or better features for all the arms. I was thus wondering if there is any equivalence between the two formulations or a way to relate them.

I find two different types of formulations for Contextual Bandits:

Definition 1: In a contextual bandits problem, there is a distribution $P$ over $(x,r_1,...,r_k)$, where $x$ is the context, $a \in [k]$ is one of the $k$ arms to be pulled, and $r_a$ is the reward for arm $a$. The problem is a repeated game: on each round, a sample $(x, r_1, ..., r_k)$ is drawn from $P$, the context $x$ is announced, and for precisely one arm $a$ chosen by the player, its reward $r_a$ is revealed.

Definition 2: The algorithm observes the current user $u_t$ and a set $A_t$ of arms or actions together with their feature vectors $x_{t,a}$ for $a \in A_t$. The vector $x_{t,a}$ summarizes information of both the user $u_t$ and arm $a$, and will be referred to as the context. Based on observed payoffs in previous trials, the agent selects an arm $a_t ∈ A_t$, and receives payoff $r_t$ whose expectation depends on both the user $u_t$ and the arm $a_t$.

When stating def. 2, the authors of the paper cite the paper from def. 1: "Following previous work" [(def. 1)] "we call it a contextual bandit." This is very confusing to me.

In particular, def. 1 assumes that only one context is revealed to the learner. In the second formulation, you observe "contexts" or better features for all the arms. I was thus wondering if there is any equivalence between the two formulations or a way to relate them.

I find two different type of Contextual Bandit problem formulations in the literature:

Definition 1: (https://hunch.net/~jl/projects/interactive/sidebandits/bandit.pdf) In a contextual bandits problem, there is a distribution $P$ over $(x,r_1,...,r_k)$, where x is context, $a \in \{1,...,k\}$ is one of the k arms to be pulled, and $r_a \in [0, 1]$ is the reward for arm $a$. The problem is a repeated game: on each round, a sample $(x, r_1, ..., r_k)$ is drawn from $P$, the context $x$ is announced, and then for precisely one arm a chosen by the player, its reward $r_a$ is revealed.

Definition 2: (http://rob.schapire.net/papers/www10.pdf) The algorithm observes the current user $u_t$ and a set $A_t$ of arms or actions together with their feature vectors $x_{t,a}$ for $a \in A_t$. The vector $x_{t,a}$ summarizes information of both the user ut and arm $a$, and will be referred to as the context. Based on observed payoffs in previous trials, A chooses an arm $a_t ∈ A_t$, and receives payoff $r_t,a_t$ whose expectation depends on both the user $u_t$ and the arm $a_t$.

The fact that when stating def. 2, the authors of http://rob.schapire.net/papers/www10.pdf cite the paper from the first definition is very confusing to me. In particular, they say "Following previous work [18], we call it a contextual bandit.".

In particular, def. 1 assumes that only one context is revealed to the learner. In the second formulation, you observe "contexts" or better features for all the arms. I was thus wondering if there is any equivalence between the two formulations or a way to relate them.

I find two different types of formulations for Contextual Bandits:

Definition 1: In a contextual bandits problem, there is a distribution $P$ over $(x,r_1,...,r_k)$, where $x$ is the context, $a \in [k]$ is one of the $k$ arms to be pulled, and $r_a$ is the reward for arm $a$. The problem is a repeated game: on each round, a sample $(x, r_1, ..., r_k)$ is drawn from $P$, the context $x$ is announced, and for precisely one arm $a$ chosen by the player, its reward $r_a$ is revealed.

Definition 2: The algorithm observes the current user $u_t$ and a set $A_t$ of arms or actions together with their feature vectors $x_{t,a}$ for $a \in A_t$. The vector $x_{t,a}$ summarizes information of both the user ut and arm $a$, and will be referred to as the context. Based on observed payoffs in previous trials, A chooses an arm $a_t ∈ A_t$, and receives payoff $r_t,a_t$ whose expectation depends on both the user $u_t$ and the arm $a_t$.

The fact that when stating def. 2, the authors of the paper cite the paper from def. 1: Following previous work (def. 1), we call it a contextual bandit. This is very confusing to me.

In particular, def. 1 assumes that only one context is revealed to the learner. In the second formulation, you observe "contexts" or better features for all the arms. I was thus wondering if there is any equivalence between the two formulations or a way to relate them.

I find two different type of Contextual Bandit problem formulations in the literature:

Definition 1: (https://hunch.net/~jl/projects/interactive/sidebandits/bandit.pdf) In a contextual bandits problem, there is a distribution $P$ over $(x,r_1,...,r_k)$, where x is context, $a \in \{1,...,k\}$ is one of the k arms to be pulled, and $r_a \in [0, 1]$ is the reward for arm $a$. The problem is a repeated game: on each round, a sample $(x, r_1, ..., r_k)$ is drawn from $P$, the context $x$ is announced, and then for precisely one arm a chosen by the player, its reward $r_a$ is revealed.

Definition 2: (http://rob.schapire.net/papers/www10.pdf) The algorithm observes the current user $u_t$ and a set $A_t$ of arms or actions together with their feature vectors $x_{t,a}$ for $a \in A_t$. The vector $x_{t,a}$ summarizes information of both the user ut and arm $a$, and will be referred to as the context. Based on observed payoffs in previous trials, A chooses an arm $a_t ∈ A_t$, and receives payoff $r_t,a_t$ whose expectation depends on both the user $u_t$ and the arm $a_t$.

The fact that when stating definition number 2, the authors of http://rob.schapire.net/papers/www10.pdf cite the paper from the first definition is very confusing to me. In particular, they say "Following previous work [18], we call it a contextual bandit.1".

In particular, in Definition 1 assumes that only one context is revealed to the learner. In the second formulation, you observe "contexts" or better features for all the arms. I was thus wondering if there is any equivalence between the two formulations or a way to relate them.

All type of suggestion will be very appreciated. Thanks a lot for your help.

I find two different type of Contextual Bandit problem formulations in the literature:

Definition 1: (https://hunch.net/~jl/projects/interactive/sidebandits/bandit.pdf) In a contextual bandits problem, there is a distribution $P$ over $(x,r_1,...,r_k)$, where x is context, $a \in \{1,...,k\}$ is one of the k arms to be pulled, and $r_a \in [0, 1]$ is the reward for arm $a$. The problem is a repeated game: on each round, a sample $(x, r_1, ..., r_k)$ is drawn from $P$, the context $x$ is announced, and then for precisely one arm a chosen by the player, its reward $r_a$ is revealed.

Definition 2: (http://rob.schapire.net/papers/www10.pdf) The algorithm observes the current user $u_t$ and a set $A_t$ of arms or actions together with their feature vectors $x_{t,a}$ for $a \in A_t$. The vector $x_{t,a}$ summarizes information of both the user ut and arm $a$, and will be referred to as the context. Based on observed payoffs in previous trials, A chooses an arm $a_t ∈ A_t$, and receives payoff $r_t,a_t$ whose expectation depends on both the user $u_t$ and the arm $a_t$.

The fact that when stating def. 2, the authors of http://rob.schapire.net/papers/www10.pdf cite the paper from the first definition is very confusing to me. In particular, they say "Following previous work [18], we call it a contextual bandit.".

In particular, def. 1 assumes that only one context is revealed to the learner. In the second formulation, you observe "contexts" or better features for all the arms. I was thus wondering if there is any equivalence between the two formulations or a way to relate them.