Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [reinforcement-learning]

A set of dynamic strategies by which an algorithm can learn the structure of an environment online by adaptively taking actions associated with different rewards so as to maximize the rewards earned.

1
vote
0answers
11 views

Reinforcement Learning partial derivative of loss function w.r.t. input of softmax

In the paper "Self-critical sequence training for image captioning" (link) on page 3 they define the loss function (of the parameters $\theta$) of an image captioning system as the negative expected ...
0
votes
0answers
4 views

Does training a VAE online from a nonstationary distribution affect convergence?

For example, using data being sampled from reinforcement learning as the policy improves. If there is an issue, how would we address the issue?
2
votes
1answer
70 views
+50

Why are rewards scaled when using Reinforcement Learning (RL) algorithms in practice?

I was going through this tutorial in pytorch and saw the following code: ...
0
votes
0answers
16 views

what is the simplest algorithm of trial and error strategy? [on hold]

I want to implement a simple trial and error algorithm to control an event by chechking after every operation if a set of input parameters change or not. I know that Reinforcement learning ...
1
vote
1answer
30 views

The proof for policy iteration algorithm's optimality

I am trying to understand why the policy iteration algorithm in Reinforcement Learning always improves the value function until it converges. Let's assume we have the policy $\pi_0(s)$ and our value ...
0
votes
0answers
20 views

Is deep reinforcement learning better or worse than state of the art multiarmed bandit methods for single step MDPs? [closed]

Any examples and links to the state of the art bandit methods would be appreciated.
2
votes
0answers
44 views

REINFORCE calculating the log policy gradient for a continuous action space

I've noticed that when modelling a continuous action space, the default thing to do is to estimate a mean and a variance where each is parameterized by a neural network or some other model. I also ...
2
votes
1answer
57 views

Understanding the temporal difference prediction error formula which uses a derivative

I'm very new to understanding the concept of prediction error underlying the output of the critic in the critic-actor method (RL learning), so bear with me, please. For the temporal difference ...
2
votes
0answers
27 views

What is the gradient of the objective function in the Soft Actor-Critic paper?

In the paper "Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor", they define the loss function for the policy network as $$ J_\pi(\phi)=\mathbb E_{...
1
vote
0answers
26 views

Best approach for learning Reinforcement Learning coming from economics?

I have an economics background so I have have Calculus, Linear Algebra, Diff. Eq., 2 semesters of Stats and Prob. and some Python Knowledge. My school offers a 2 months postgraduate course in ...
1
vote
0answers
17 views

When using linear function approximation how (and why) should I incorporate the actions into the feature vector?

When reading R. Sutton: Reinforcement Learning - An Introduction (2nd edition), in chapter 10.1 Episodic Semi-gradient Control, the Mountain Car problem is mentioned and as an example it is solved ...
2
votes
0answers
16 views

Proof that the addition of a baseline to the REINFORCE algorithm reduces the variance

A widely used variation of REINFORCE is to subtract a baseline value $b$ from the return $G_t$ to reduce the variance of gradient estimation, such that \begin{align} \nabla_\theta J(\theta) & \...
3
votes
0answers
45 views

Why does the Policy Gradient Theorem generalize to continuous action spaces

The policy gradient is generally in the shape of the following: $$ L^{PG}(\theta) = \mathbb{E}_t \left[ \log \pi_\theta(a_t \mid s_t) A_t \right] $$ Where $\pi$ represents the probability of taking ...
0
votes
1answer
29 views

Can you implement Replay Buffers for Reinforcement Learning when most experiences give zero reward?

Specifically, for a deep deterministic policy gradient, DDPG, to expedite the learning speed, it's recommended to use a Replay Buffer What if the reward is only given at a terminal state? Or, most of ...
1
vote
0answers
20 views

Vanishing reward function in Q-Learning

Imagine that the agent receives a positive reward upon reaching a state $s$. Once the state $s$ has been reached the positive reward associated with it vanishes and appears somewhere else in the state ...
1
vote
1answer
23 views

Transfer with different sized state spaces in neural networks/deep reinforcement learning

Say we are transferring sequentially from environment 1-3 below, where the text corresponding to each environment describes its observation space. Env 1 observation: position of robot Env 2 ...
0
votes
1answer
38 views

Variance of reparameterization trick and score function

For a function $\mathbf E_{z\sim q_\phi(z|x)}[f(z)]$(assuming $f$ is continuous), where $q_\phi$ is a Gaussian distribution, if we want to compute the gradient w.r.t. $\phi$, we have two way to do ...
1
vote
0answers
20 views

How to use trial and error algorithm to predict the next number in a sequence?

I have a time series data. I want to use trial and error algorithms to predict the next number in a variation_sequence. I mean about Trial and error algorithm is using an online learning and where I ...
1
vote
0answers
31 views

Natural way to construct stochastic policy from value function?

I've been thinking about this question for a while now and I can't seem to convince myself of the (in)validity of deriving a probabilistic policy from a state-action value function $Q(s,a)$. Of ...
1
vote
0answers
28 views

How to redistribute probabilities based on feedback?

Imagine we have a set of pieces of information and we don't know anything about them. We take a single piece randomly and decide if it is useful or not. Then the probabilities of each piece to be ...
1
vote
1answer
56 views

What does this notation $p(s', r \mid s, a)$ mean in reinforcement learning?

I was reading a book on reinforcement learning, and came across the following notation: Possibly stupid question, but I cant seem to google how to read this. Is it A) Probability of (s' and r) GIVEN ...
0
votes
1answer
12 views

How to set up a Q function approximator using neural net for DDPG?

For discrete action space, I thought the "conventional method" is to set up the neural network in such a way that the inputs are the states and each of the output node represents possible action with ...
2
votes
0answers
30 views

How to identify too small network in reinforcement learning?

I am using Open AI's code to do a RL task on an environment that I built myself. I tried some network architectures, and they all converge, faster or slower on CartPole. On my environment, the ...
0
votes
0answers
14 views

Is it possible to extend the near-optimal finite horizon average reward to infinite horizon discounted reward in RL?

Is it possible to extend the near-optimal finite horizon average reward to near-optimal infinite horizon discounted reward in RL (for example, in the context of Q-learning)? If yes, how? I believe ...
0
votes
1answer
25 views

What does environment dynamic means in Reinforcement learning

From the book Reinforcement learning: an introduction, I have two questions: 1) there is the following sentence: "If the environment's dynamics are completely known, then finding the optimal policy ...
0
votes
1answer
12 views

Confusion about the derivation of the TD-Learning update rule

I am currently trying to understand the paper "Learning to Predict by the Methods of Temporal Differences" by Sutton. I am stuck with the following step: (From "Learning to Predict by the Methods of ...
1
vote
1answer
40 views

Are there frequentist approaches to Thompson Sampling?

What is the theoretical reason why Thompson Sampling needs to involve posterior distributions? Why can we not sample over predictive distributions? (or is the issue that predictive frequentist ...
0
votes
0answers
18 views

Why does is make sense to normalize rewards per episode in reinforcement learning? [migrated]

In Open AI's actor-critic and in Open AI's REINFORCE, the rewards are being normalized like so rewards = (rewards - rewards.mean()) / (rewards.std() + eps) ON ...
0
votes
1answer
36 views

How to explain and visualize a Q Learning Agent?

What are some common visualization approaches used in explaining the behavior of a Q-Learning agent? Here is an excerpt of some example Q values for 5 actions serialized to json: ...
1
vote
1answer
25 views

Multi armed bandit algorithms failing with un-scaled rewards

I am experimenting with the multi-armed bandit algorithms (namely: epsilon greedy, decaying epsilon greedy, optimistic initial value, upper confidence interval, and Thompson sampling). My reward is ...
1
vote
1answer
47 views

DQN agent helped by a prediction model

Suppose I have a regression model that can make predictions on stock price movements for 10 steps ahead. The labels are ...
0
votes
1answer
28 views

Reinforcement Learning: Actor Critic - Why is weight sharing possible?

I was looking at Open Ai's actor-critic code and noticed that some of the neural network's weights are shared ...
1
vote
0answers
51 views

Reward function for intraday trading [duplicate]

I am working to build an deep reinforcement learning agent which can place orders (i.e. limit buy and limit sell orders). The actions are ...
0
votes
1answer
89 views

A reward becomes a penalty if

I am working to build a reinforcement agent with DQN. The agent would be able to place buy and sell orders for a day trading purpose. I am facing a little problem with that project. The question is "...
0
votes
1answer
14 views

How does DQN parameter updates work in simulation?

I've already read almost every Questions-answers and material related to DQN, deep reinforcement learning, but I'm struggling to start working on simulation. First of all, I'm trying to code using ...
2
votes
0answers
72 views

Confusion about step in deriving Bellman equation from value function

I am reading Reinforcement Learning, An Introduction by Sutton, Barto and I came across the derivation $$ \begin{align} v_{\pi}(s) &= \mathbb{E}_{\pi}\left[ G_{t} | S_{t} = s \right] \\ &= \...
2
votes
0answers
25 views

Deep Q - Learning Exploration - BestQ Value

I am trying to implement a Deep Q - Network to play Asteroids. Unfortunately, I am not sure how to calculate the Q Value exactly if I am exploring. For example, the Agent is exploring for 1 second (...
1
vote
0answers
27 views

Initialize replay memory and action value function Q

I am not sure I can ask that question here, but I will try an attempt. I am trying to implement Beat Atari with Deep Reinforcement Learning. They explained very well each steps, but they ask you to ...
0
votes
1answer
38 views

How to initialize and update Q network weights in DQN with delayed/sparse reward

I cannot wrap my head around one thing about a delayed/sparse reward reinforcement learning. In my problem I would like to teach a neural network to be able to play a simple two-player game. Or in ...
0
votes
0answers
24 views

Why isn't my reinforcement learning agent learning anything useful?

I've been trying to implement a Q-learning agent to play the game of snake. There are many examples of deep Q-learning agents doing this on github but I couldn't find any with simple Q-learning and as ...
0
votes
0answers
63 views

How to apply Reinforcement Learning to time-series sampling?

I want to apply the concept of Reinforcement Learning (RL) to help an agent decide when to sample an unobserved time-series signal. Can you help/guide me on how to proceed? The problem setup can be ...
0
votes
1answer
43 views

How to apply multi agent deep reinforcement learning to an environment with discrete action space

Do you know or have heard about any cutting edge deep reinforcement-learning algorithm which can be successfully applied for discrete action-spaces in multi-agent settings? I have been researching ...
1
vote
2answers
116 views

What does a Diagonal Gaussian Distribution look like in 3 dimensions?

I was able to find https://brilliant.org/wiki/multivariate-normal-distribution/ and I am aware the the diagonal gaussian distribution is a special case where the only entries are on the diagonal, ...
2
votes
1answer
52 views

convergence of an algorithm [closed]

I want to know when we speak about the convergence of an algorithm, what are the conditions that we should check. For example, I was looking for the convergence of the policy iteration algorithm in ...
1
vote
2answers
23 views

How can policy parameterization be simpler than action-value parameterization in function approximation?

In the second edition of the book "Reinforcement Learning: an introduction" by Sutton and Bato page 323 (Policy gradient chapter) it says that: "Perhaps the simplest advantage that policy ...
0
votes
0answers
26 views

Incremental/Online Learning for Reinforcement Learning

I have been reading the book Reinforcement Learning: An Introduction by Sutton and Barto, and I have some questions about extending the value function approximation to non-differential functions: Is ...
1
vote
0answers
15 views

Convergence criterion for R-learning algorithm

I'm trying to find a policy for a simple game using R-learning algorithm. I have a field with values (agent can move in 4 directions) and the goal is to get from starting point to finish point with ...
0
votes
0answers
17 views

Will Multi-Objective Reinforcement Learning be overkill to this problem?

Consider the situation on sports trading. Suppose I take a back bet on team A, at price 1.8 with stake \$10, this would result a potential profit of \$8 or a potential loss \$10. Then I take a lay bet ...
0
votes
0answers
83 views

policy iteration convergence

There is a question here for 2014 about the convergence of policy iteration algorithm with two answers > Question However, it is not clear for me how we change the value functions after one policy ...
4
votes
2answers
121 views

Value iteration does not converge when using Q learning

I have a simple game and want my agent to play it with a help of reinforcement learning. We have a board and a value in each cell. The goal is to go from start to finish point with the highest score (...