Questions tagged [contextual-bandit]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
5
votes
1answer
89 views

Equivalence of Contextual Bandit formulations

I find two different type of Contextual Bandit problem formulations in the literature: Definition 1: (https://hunch.net/~jl/projects/interactive/sidebandits/bandit.pdf) In a contextual bandits problem,...
0
votes
0answers
13 views

Linear Thompson sampling - non linear relations dominate sampled rewards

I'm implementing linear Thompson sampling for a project. I am simulating random features with rewards that are linear related to some features them and non linear to others. For the non linear ...
0
votes
1answer
34 views

What Does Oracle in Multi-Armed Bandit Literatures?

Often encounter "oracle" this term in multi-armed bandit literatures. But none of papers explain what that means. An example: Practical Contextual Bandits with Regression Oracles
2
votes
1answer
52 views

How to Deploy Contextual Bandits in Online Experimentation Platform?

This question is about how to deploy contextual bandits(CMAB) in the context of web site optimization and online experimentation. I implemented contextual free MAB(MAB). When I run a MAB experiment, I ...
0
votes
1answer
34 views

Mechanism of Adversarial Multi-Armed Bandit Problem?

I am studying the Bandit Algorithms book by Tor Lattimore and Csaba Szepesv´ari and I have studied the adversarial bandit problem. However, I don't understand what is the mechanism of adversarial ...
2
votes
2answers
75 views

Contextual bandits: Number of models to estimate

I have recently read several papers on contextual bandits especially for the case of binary rewards. However, one very basic aspect is not entirely clear to me: In some papers (e.g. here https://arxiv....
1
vote
0answers
42 views

off-policy evaluation in reinforcement learning

IPS estimator, which is used for off-policy evaluation in a contextual bandit problem, is well explained here: Doubly Robust Policy Evaluation andOptimization https://arxiv.org/pdf/1503.02834.pdf The ...
2
votes
0answers
69 views

How can one optimize black-box functions given context?

Libraries like hyperopt or scikit-optimize allow one to optimize a black-box function. However, they do not allow specifying contextual information outside of the parameters to be chosen by the ...
0
votes
1answer
45 views

Why is having low variance important in offline policy evaluation of reinforcement learning?

Intuitively, I understand that having an unbiased estimate of a policy is important because being biased just means that our estimate is distant from the truth value. However, I don't understand ...
0
votes
1answer
30 views

way to transform reinforcement learning problems to bandit problems

I wonder what a general way looks like to transform reinforcement learning problems to bandit problems (especially contextual bandit problems) Thank you!
2
votes
1answer
122 views

Using IPS(inverse probability weighting) with a deterministic policy as the logging policy

In a contextual bandit problem, why can't we use inverse probability weighting (inverse propensity score) with a deterministic policy as the logging policy? Could you give me a concrete example?
0
votes
1answer
68 views

Policy evaluation in contextual bandit setting

I am currently reading a paper whose links is (Exploration Scavenging) http://delivery.acm.org/10.1145/1400000/1390223/p528-langford.pdf?ip=128.135.98.49&id=1390223&acc=ACTIVE%20SERVICE&...
0
votes
1answer
68 views

quick questions about a contextual bandit problem

I am currently reading the paper "Learning from Logged Implicit Exploration Data" https://arxiv.org/pdf/1003.0120.pdf. But I believe the questions I have can be answered without reading the whole ...
0
votes
1answer
181 views

How do we calculate regret or know best action in a multi armed bandit at run time in a program(python)?

Consider $K$ arms, each having a normal distribution with mean $\mu_k$ taken from: $$\mu_k ∼ \mathbb{N}(0,1)$$ Then, the reward function $R_t(\mu_k)$ at time $t$ has distribution: $$R_t(\mu_k) ∼ \...
0
votes
1answer
90 views

large variance from inverse probability weighting (inverse propensity score)

I heard if the observed data that will be used in the inverse probability weighting method is too small, the estimator based on the weighting will have a large variance. Could you explain why that is ...
1
vote
0answers
35 views

EXP4 algorithm: Changes in exponential weight update rule for multiple arms?

I am new to contextual MAB problems so I have a few doubts. I am trying to implement the EXP4 algorithm, however in EXP4 we only choose one arm at a time based on the advice of the Experts. We then ...
1
vote
0answers
67 views

Is LinUCB Hannan consistent?

I'm going through the textbook Bandit Algorithms, by Lattimore and Szepesvari (http://downloads.tor-lattimore.com/banditbook/book.pdf). It describes regret bounds for the LinUCB algorithm of the form:...
2
votes
0answers
27 views

Why psuedo regret and not regret is used in adversarial bandits?

In adversarial settings, psuedo regret and not the actual regret is used. The explanation I have been given is that with actual regret the problem is no longer learnable (that is adversary can ...
1
vote
1answer
1k views

In a multi-arm bandit problem, how does one calculate the cumulative regret in real life?

I was recently looking at some information on Multi-arm Bandits, and they have the notion of cumulative regret. It is defined as the following: Consider $K$ arms, each having a normal distribution ...
1
vote
0answers
546 views

Contextual bandits vowpal wabbit cost and training

I have 3 different methods of showing recommendations of products to users. I want to use vowpal wabbit to find context specific policies to choose the optimal action (3 actions as there are 3 methods ...
2
votes
1answer
258 views

For a Multi-arm Bandits set-up, does the Signal to Noise Ratio have any meaning there?

Suppose that we have a Multi-arm Bandit set-up with $K = 5$ bandits. Each bandit has a reward distribution of: $$ X_i \sim Bern(p_i), \ \ \ i \in \{1, \ldots, 5\} $$ In literature about MABs, ...
1
vote
1answer
621 views

EXP4 algorithm for contextual bandits: where do experts come from?

I am working on an implementation of the EXP4 algorithm in the context of a pricing decision (e.g. given a context, the user should be given a price from a few pre-determined options). The EXP4 uses "...
1
vote
1answer
50 views

Posterior sampling for bandit

I am looking at this paper on posterior sampling. The algorithm is on page 8 (image below): Let’s say I have 3 arms and on line 22 arm 3 is the best followed by arm 2 then arm Line 24 calculates the ...
0
votes
0answers
52 views

Name of “statistical” method

This might be a too specific question, but I have no idea where else to turn to. I'm creating Multi-Armed Bandits with different approaches. One approach has been to create a randomized matrix and ...
3
votes
1answer
356 views

Multi-armed bandit in face of full reward information

I am new to this area of machine learning. I am just walking myself through UCB1 algorithm which seems to assume that the payoff can be learnt only for action that ...
3
votes
2answers
417 views

Exploiting features in a multiarmed bandit scenario

I am facing a challenging problem: Say I have shirts of three different colors (same price). And say I am running a strange kind of store in into which people come in one by one, and I can show ...
14
votes
1answer
1k views

Cost functions for contextual bandits

I'm using vowpal wabbit to solve a contextual-bandit problem. I'm showing ads to users, and I have a fair bit of information about the context in which the ad is shown (e.g. who the user is, what ...