Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [reinforcement-learning]

A set of dynamic strategies by which an algorithm can learn the structure of an environment online by adaptively taking actions associated with different rewards so as to maximize the rewards earned.

0
votes
0answers
5 views

Convergence criterion for R-learning algorithm

I'm trying to find a policy for a simple game using R-learning algorithm. I have a field with values (agent can move in 4 directions) and the goal is to get from starting point to finish point with ...
0
votes
0answers
8 views

Will Multi-Objective Reinforcement Learning be overkill to this problem?

Consider the situation on sports trading. Suppose I take a back bet on team A, at price 1.8 with stake \$10, this would result a potential profit of \$8 or a potential loss \$10. Then I take a lay bet ...
0
votes
0answers
28 views

policy iteration convergence

There is a question here for 2014 about the convergence of policy iteration algorithm with two answers > Question However, it is not clear for me how we change the value functions after one policy ...
1
vote
1answer
37 views

Value iteration does not converge when using Q learning

I have a simple game and want my agent to play it with a help of reinforcement learning. We have a board and a value in each cell. The goal is to go from start to finish point with the highest score (...
1
vote
1answer
32 views

Two different value functions formulations, how can I reconcile them?

I've been studying Reinforcement learning for the past months, and using different sources, I've been seeing different formulations for the same thing. Specifically for value iteration: My question ...
2
votes
1answer
38 views

Negative Rewards in Reinforcement Learning

I'm using self play to teach a model to play games (e.g. board games). Basically the model is playing against itself. When playing as player 2 I switch the perspective as if it is player 1 to train ...
0
votes
0answers
25 views

Why Monte Carlo sampling is not needed for reparameterization trick?

To esitimate $\nabla_\theta \mathbb{E}_{z\sim p_\theta(z)}[f(z)]$, we have two options: REINFORCE: $\nabla_\theta \mathbb{E}_{z\sim p_\theta(z)}[f(z)] = \mathbb{E}_{z\sim p_\theta(z)}[ f(z)\nabla_\...
1
vote
1answer
30 views

Feature Scaling in Reinforcement Learning

I am working with RL algorithms like DQN and ActorCritic and I'm curious whether there is a way to correctly scale features which represent state or state/action pair while learning parameters of ...
1
vote
0answers
24 views

Reinforcement Learning - What is the logic behind actor-critic methods? Why use a critic?

Following David Silver's course, I came across the actor-critic policy improvement algorithm family. It holds For one-step Markov decision processes that $$\nabla_{\theta}J(\theta) = \mathbb{E}_{\...
0
votes
0answers
21 views

Value Iteration not learning anything usefull

I am currently trying to learn the optimal policy for a derivation of the Open AI gym environment "Pendulum-v0". I know there are more suited methods for continous domains, but we are doing a somehow ...
1
vote
2answers
23 views

actions with no effect on the enviroment in RL

A somewhat naive question about MDP and RL: Usually one assumes that the next environment state depends on the action the agent chooses. E.g. this is clearly the case in games like go. But what if ...
0
votes
1answer
18 views

State space and latent space instability

Reinforcement learning assumes an MDP with an a priori state space representation. Assume the state space is the raw images from a game, and we use convNets or another method to generate s latent ...
0
votes
1answer
25 views

How can policies be ordered in reinforcement learning?

Following Sutton, Barto "Reinforcement Learning: An Introduction", in 3.6 Optimal Policies and Optimal Value Functions they define an ordering between policies: A policy $\pi$ is defined to be ...
0
votes
0answers
39 views

Machine Learning for Strategic Planning Simulation

I am student working on a project for university, but I am kinda stuck right now. I had this idea to use my recently acquired knowledge in AI to build a model that is capable of mastering a strategic ...
2
votes
1answer
57 views

How to handle a changing action space in Reinforcement Learning

I'm training a Reinforcement Model playing a game with self learning.(A second instance is its opponent). An agent has a set of possible action to choose from in each state. Those actions usually ...
1
vote
0answers
28 views

DeepMind RMSprop vs. Tensorflow RMSprop

Some neural network architectures work better with RMSprop than e.g. ADAM. So for example stated by DeepMind in their work with Atari games and reinforcement learning. Maciej Jaskowski reproduced the ...
0
votes
0answers
15 views

Why the loss function of Vanilla Policy Gradient cannot tell how good the model is?

When reading Spinning Up (the gray box titled "You Should Know" in the middle of the page), it says the loss function does not indicate how well the model does. I cannot figure out why. I think the ...
0
votes
0answers
25 views

Reinforce Algo - Theta parametrized by a deep NN

I'm following a course on Reinforcement learning on EDX called "Reinforcement Learning explained - DAT257x" . I m learning about policy gradient method and in particular about REINFORCE algo. At a ...
1
vote
1answer
42 views

Monte Carlo $\epsilon$ - greedy policy is better than $\epsilon$- soft policy

In the RL book of Barto and Sutton, the authors have proved that any $\epsilon$-greedy policy with respect to $q_{\pi}$ is an improvement over any $\epsilon$-soft policy $\pi$ is assured by the policy ...
0
votes
0answers
20 views

need help understanding the benefit of score function estimator

The score function estimator a.k.a REINFORCE policy gradient in reinforcement learning is (from http://blog.shakirm.com/2015/11/machine-learning-trick-of-the-day-5-log-derivative-trick/): \begin{...
2
votes
1answer
26 views

What do the mu and (.) symbol represent in the deterministic/policy formula?

I am currently going through the OpenAI Spinning Up documents and the following notation puzzled me. What do the mu and . sign mean in the following two formulas?
0
votes
0answers
20 views

Agents playing against each other (Reinforcement Learning)

To train my reinforcement model to play a two player game I could either play by myself or let it play against a second instance of itself. Are there any drawbacks which I have to take care of? Some ...
1
vote
0answers
44 views

Reinforcement Learning and Introduction - Chapter 3 - Continuing vs Episodic Tasks

The exercise 3.5 in the book "Reinforcement Learning - an introduciton" asks for a modified version of eq (3.3) for the episodic tasks. The Equation (3.3) states that, considering all $s'$ and $r$ on ...
0
votes
1answer
15 views

Generalization of different play-field sizes

I'm currently working on a Deep Reinforcement Learning model for the game "connect 4". Before I started I read some rules and facts about "connect 4". Thats when I noticed after years of playing it, ...
2
votes
1answer
34 views

Why in Q-Learning, policy pi is evaluated through another policy u?

I've been watching David Silver's courses about Reinforcement Learning. According to his lectures, policy $\pi$ is evaluated by evaluating another policy $\mu$. But I cannot understand: why is it so? ...
0
votes
1answer
28 views

Decision making in different intervals in MDPs

I want to model a problem as an MDP model where every day is divided into small time slots (for example minutes) and two decisions A and ...
3
votes
1answer
103 views

If the Markov assumption is wrong, will a learner still converge to a stable policy?

I'm trying to figure out what guarantees can be made if a learner wrongly assumes a problem obeys the Markov transition property. Assume I have a problem defined by a partially observable Markov ...
1
vote
1answer
13 views

Clarification regarding Markov Decision Process (MDP) formulation

Most of the reinforcement learning problems are dealt with using an MDP framework. I’m a bit confused about the formulation after reading the paper: https://arxiv.org/abs/1503.02244 In an continuous ...
1
vote
1answer
32 views

Can some one explain me what is difference between Markov process and Markov Decision Process

Markov Process : A stochastic process has Markov property if conditional probability distribution of future states of process depends only upon present state and not on the sequence of events that ...
3
votes
2answers
106 views

Principled understanding of why AlphaZero's algorithm works?

Roughly, AlphaGo/AlphaGo Zero 's algorithm is as follows: Using a policy network, generate a distribution of move probabilities (intuitively, capturing how good those moves are based on a first-...
2
votes
2answers
47 views

Why gradient descent of log probabilities in Alpha Go?

In the original AlphaGo paper, it is stated that the policy network is trained with the following gradient: I don't understand why this gradient makes sense. Why do we want to move the parameters ...
0
votes
0answers
15 views

For reinforcement learning, how to solve wrong recording issue when learning from expert?

Suppose I want to build an AI to play a game, I play it first as the expert for AI to record. But there are some issues: For adjacent frames, for example, the actions are →,→→,when recorindg , how ...
1
vote
1answer
33 views

How to minimize sharpe ratio with LSTM recurrent neural network?

I've read some articles about trading using recurrent reinforcement learning such as this one. The point where I do not fully understand is how to construct the cost/loss function. In the article, ...
2
votes
1answer
29 views

What does an expectation with respect to a policy mean in the reinforcement learning value function

I would like to know what the formal definition of the following expression is $$ V_\pi(s) = \mathbb{E}_{\pi}(G_{t+1} | S_t =s) $$ What does it mean to have the policy in the subscript? How would I ...
0
votes
1answer
47 views

what is the meaning or intuition of entropy (from the point of view of reinforcement learning)

Can someone give an intuition of the concept 'entropy'? I am reading maximum entropy inverse reinforcement learning and I wanted to ask what the meaning intuition of 'entropy' is. I understand ...
0
votes
0answers
39 views

Adding additional constrains to OpenAi Gym

I'm currently working trough some examples which should finally end in a DQN Reinforcement Learning for the CartPole example in the openAI-Gym. Copied some code from GitHub which isn't deep yet: <...
1
vote
1answer
16 views

How to model episodic task with pre-determined total time?

I want to model a problem as an MDP and solve it with reinforcement learning algorithms. Suppose that the problem is episodic and it finishes at some point. However, termination occurs if either the ...
1
vote
1answer
37 views

Trouble understanding value iteration

I have trouble understanding how the value iteration algorithm for MDP:s work. I'm trying to follow the canonical grid world example (slide 17), but I don't get the correct results. Here's my work: ...
0
votes
0answers
14 views

How to implement discontinuous action for reinforcement learning?

I am training a RL agent to operate a device and the agent has a continuous action space (DDPG). The device can be off or operate in a voltage range 6-12 V. If a naively map 6-12 V to the interval [...
1
vote
0answers
20 views

High variance of returns using Asynchronous Actor-Critic Agents (A3C) on CartPole [closed]

I ran the code from the Tensorflow blog with modified running average function (it takes a running mean of the last 3 episodes only) and notice strange behavior. Although the model achieves episode ...
0
votes
1answer
45 views

How state aggregation in reinforcement learning works?

I am watching Prediction with linear approximation video course in the RL class by prof. Sutton. He presented state aggregation approach on a random walk problem. It seems that this approach just ...
1
vote
1answer
90 views

Convergence of Value Iteration in Reinforcement Learning

I am struggling to understand when the value iteration algorithm converge. Suppose that I use a discount 1. Will the algorithm always converge if I have terminal states in the MDP or it has to do on ...
1
vote
1answer
95 views

Value Iteration For Terminal States in MDP

I am a little bit confused regarding the value iteration algorithm. When I loop over states should I visit the terminal states, if any, or not? In Sutton's book on page 83 it says that we do not ...
2
votes
1answer
31 views

Number of states in Reinforcement Learning

I'm having trouble understanding the concept of states in RL. The Policy maps an action to a state. I'm thinking about the state as clearly defined situation. E.g. in connect four assuming a 8x8 ...
0
votes
0answers
11 views

Alternative to causal graphs for representing spatial structure in a markov process?

This question is about how to formalize a particular structure of MDP's in an AI/Machine learning context. Consider a markov decision process in a reinforcement learning context. Causal graphs can be ...
1
vote
0answers
20 views

Reinforcement learning for segmenting robot's path to reflect the true distances

I've a grid of rectangles acting as blocks. The robot traverses through the inter-spaces between these consecutive blocks. Now I have sensor data streaming in representing Right and left wheel speeds. ...
0
votes
0answers
42 views

Prioritized Replay: How does the rank-based prioritization work out?

In the paper "Prioritized Experience Replay", the authors introduced a rank-based way to compute the priority of a transition. They said For the rank-based variant, we can approximate the ...
0
votes
0answers
16 views

Approximate reinforcement learning vs approximate dynamic programmin?

I know that dynamic programming uses the model of the environment while many RL methods are model-free. However, I am willing to know the difference between ADP and ARL and I would be thankful if ...
0
votes
2answers
51 views

Suitable project for reinforcement learning [closed]

To get more insight in reinforcement learning I'm looking for a project for. I thought about the game connect four. Doing some research about existing works with RL and connect four I found out that ...
1
vote
1answer
38 views

Derivation of Monte Carlo Policy Gradient for REINFORCE

On page 270 of this draft of Intro to Reinforcement Learning by Sutton and Burton, the authors simplify the policy gradient as follows: Since the action-value function equals the conditional ...