Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

A set of dynamic strategies by which an algorithm can learn the structure of an environment online by adaptively taking actions associated with different rewards so as to maximize the rewards earned.

0
votes
0answers
7 views

How to implement discontinuous action for reinforcement learning?

I am training a RL agent to operate a device and the agent has a continuous action space (DDPG). The device can be off or operate in a voltage range 6-12 V. If a naively map 6-12 V to the interval [...
1
vote
0answers
10 views

High variance of returns using Asynchronous Actor-Critic Agents (A3C) on CartPole [on hold]

I ran the code from the Tensorflow blog with modified running average function (it takes a running mean of the last 3 episodes only) and notice strange behavior. Although the model achieves episode ...
0
votes
0answers
19 views

Bellmans Equation in Sutton and Barto

If you haven't looked at Sutton before please ignore this question, I have not explained every aspect of the notation I've always been a bit confused by the derivation of Bellman equations for the ...
0
votes
1answer
22 views

How state aggregation in reinforcement learning works?

I am watching Prediction with linear approximation video course in the RL class by prof. Sutton. He presented state aggregation approach on a random walk problem. It seems that this approach just ...
1
vote
1answer
18 views

Convergence of Value Iteration in Reinforcement Learning

I am struggling to understand when the value iteration algorithm converge. Suppose that I use a discount 1. Will the algorithm always converge if I have terminal states in the MDP or it has to do on ...
1
vote
1answer
26 views

Value Iteration For Terminal States in MDP

I am a little bit confused regarding the value iteration algorithm. When I loop over states should I visit the terminal states, if any, or not? In Sutton's book on page 83 it says that we do not ...
0
votes
1answer
12 views

CartPole: Using sigmoid and softmax cause program converge differently [closed]

I am playing with the CartPole problem. It works but when I switch from Sigmoid to Softmax at the end of the network, as input for multinomial distribution, the program behaves quite differently. ...
2
votes
1answer
21 views

Number of states in Reinforcement Learning

I'm having trouble understanding the concept of states in RL. The Policy maps an action to a state. I'm thinking about the state as clearly defined situation. E.g. in connect four assuming a 8x8 ...
0
votes
0answers
9 views

Alternative to causal graphs for representing spatial structure in a markov process?

This question is about how to formalize a particular structure of MDP's in an AI/Machine learning context. Consider a markov decision process in a reinforcement learning context. Causal graphs can be ...
1
vote
0answers
18 views

Reinforcement learning for segmenting robot's path to reflect the true distances

I've a grid of rectangles acting as blocks. The robot traverses through the inter-spaces between these consecutive blocks. Now I have sensor data streaming in representing Right and left wheel speeds. ...
0
votes
0answers
29 views

Prioritized Replay: How does the rank-based prioritization work out?

In the paper "Prioritized Experience Replay", the authors introduced a rank-based way to compute the priority of a transition. They said For the rank-based variant, we can approximate the ...
0
votes
0answers
14 views

Approximate reinforcement learning vs approximate dynamic programmin?

I know that dynamic programming uses the model of the environment while many RL methods are model-free. However, I am willing to know the difference between ADP and ARL and I would be thankful if ...
0
votes
2answers
41 views

Suitable project for reinforcement learning [closed]

To get more insight in reinforcement learning I'm looking for a project for. I thought about the game connect four. Doing some research about existing works with RL and connect four I found out that ...
1
vote
1answer
18 views

Derivation of Monte Carlo Policy Gradient for REINFORCE

On page 270 of this draft of Intro to Reinforcement Learning by Sutton and Burton, the authors simplify the policy gradient as follows: Since the action-value function equals the conditional ...
1
vote
0answers
21 views

Relation between v(s) and q(s,a) in a Markov Decision Process?

I was solving questions related to backup diagrams from Reinforcement Learning: An Introduction by Barto and Sutton. Are these 4 equations mathematically correct ? Are there any shortcomings in terms ...
3
votes
1answer
31 views

Multi-agent actor-critic MADDPG algorithm confusion

I am trying to understand the paper from openAI called Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments In the paper, they mention that they combat the problem of environment ...
2
votes
0answers
16 views

POMDP books/lecture notes/tutorials

I'm looking for good references to learn more about POMDPs, preferably from a more mathematical stand point. The only good reference I've been able to find so far is: http://www.cs.toronto.edu/~...
1
vote
2answers
34 views

How can we program Reinforcement learning without transition probability and rewards?

I would like to design the optimal task distribution system using Reinforcement learning. The best advantage of Reinforcement learning compared to traditional Dynamic programming is that it is not ...
1
vote
1answer
34 views

Does a policy depend on the past?

I am wondering about the mere definition of the word 'policy'. Let us sassume that we have a finite space of states $S$ and a finite set of actions $A$. People tend to write that it is a 'stochastic ...
1
vote
0answers
23 views

LSTM Training loss decreases and increases

I am new to LSTM and deep learning. I have 3000 reviews which I am trying to train on gensim pretrained model via word embedding. I have the following model where ...
15
votes
1answer
929 views

Is supervised learning a subset of reinforcement learning?

It seems like the definition of supervised learning is a subset of reinforcement learning, with a particular type of reward function that is based on labelled data (as opposed to other information in ...
2
votes
2answers
39 views

Bilinear and K-Mapping Basis in Linear Algebra

I am reading a machine learning paper which has some mathematical terminologies that are proving a little hard for me to understand. I am going to write the lines from the papers here. The policy ...
0
votes
1answer
30 views

Estimate number of objects in an environment given agent's observations

I'm solving a reinforcement learning-like problem, where I have an agent trying to survive in a 2D room. These room contains a finite and constant number of moving objects that interact with an agent. ...
1
vote
1answer
22 views

Reinforcement Learning: definition of expected discounted return in Sutton and Barto's book

I am going through Sutton and Barto's book on reinforcement learning http://incompleteideas.net/book/bookdraft2017nov5.pdf In the book pg 44 equation 3.8, the authors define expected discounted ...
1
vote
1answer
26 views

Maximizing cross entropy reinforcement learning

I have read that in reinforcement learning, maximizing the entropy enables the policy to behave more randomly. My question comes in three parts: (1) In the equation below in the cross-entropy term ...
0
votes
0answers
26 views

Implementing RNN policy gradient in pytorch

I want to train a recurrent policy gradient which predicts action probabilities based on prior environment states. However, I am unable to backpropagate during the "update policy" step, in which the ...
0
votes
0answers
30 views

Action space for Reinforcement Learning implementation

I am confused about how to set action space for my application for which I like to use Reinforcement Learning to select the best instance. I have two groups of instances. Group_a and Group_b, ...
1
vote
0answers
43 views

Generalized Advantage Estimation

I have three questions regarding the paper High-Dimensional Continuous Control Using Generalized Advantage Estimation: 1) The authors define $\gamma$-just estimators below. My question is: Does the ...
0
votes
1answer
48 views

How to move reinforcement learning model into production?

I have trained reinforcement learning agent on a custom environment using the DQN technique. The custom environment is a simulation of a real production environment. Now I have trained NN model with ...
2
votes
1answer
77 views

Proof that an epsilon greedy policy w.r.t. $q$ values is better than the original policy $\pi$?

I was trying to understand the proof why policy improvement theorem can be applied on epsilon-greedy policy. The proof starts with the mathematical definition - I am confused on the very first line ...
1
vote
0answers
15 views

In a finite unichain average-reward MDP, how does optimal bias vector depend on stage reward

Consider a finite unichain MDP with stage reward $r$, state space $S=\{1, \dots, n\}$, action space A, and transition probability $p$. The Bellman equation is $$ h(i) + g= \max_{a \in A} ( r(i,a ) + \...
0
votes
1answer
50 views

What is a good machine learning academic research workflow/tools? [closed]

I'm interested in ML research and want to get some insights into how its done in practice. In particular: What kind of data are you dealing with? How are you dealing with hyper-param optimization? ...
1
vote
1answer
68 views

understanding natural policy gradient

I'm reading this paper on Natural Policy Gradient https://papers.nips.cc/paper/2073-a-natural-policy-gradient.pdf and have some questions regarding how it works. I'm coming at this from an ML ...
4
votes
1answer
31 views

Understanding Q-learning for continuous actions

I am reading the paper on Normalized Advantage Functions for continuous Q-learning and I am having trouble understanding why the advantage function takes this particular form: Why is the Advantage ...
1
vote
0answers
47 views

Explanation of overshooting accumulated reward in reinforcement learning

We are curious about the trend of the total accumulated reward in RL applications. Especially when it comes to an overshooting in the signal at the beginning of the training as shown in the plots from ...
2
votes
1answer
48 views

What's done with the expectations in this proof?

This is a proof of the per-decision importance sampling (theorem 1) from the appendix of: https://www.google.co.uk/url?sa=t&source=web&rct=j&url=http://scholarworks.umass.edu/cgi/...
3
votes
1answer
117 views

Why is this the correct formula to update the NN weights in Q-learning?

I'm trying to implement Q-learning to train an AI bot to play Pokemon battles. Since there is a large state space (corresponding to all possible states a battle can have in between moves), I can't use ...
0
votes
1answer
30 views

Is Reinforcement Learning suitable for optimal control problems in which actions influence rewards but not states?

In particular, rewards $r = f(s, a, s')$, but states are independent of actions $s' = g(s)$. A example could be asset trading that actions (long, short, hold) of a small trader won't affect market ...
1
vote
1answer
36 views

Policy gradient derivation confusion

In the derivation of the policy gradient, I am confused about why the sum of rewards, $r$, is constant with respect to $\theta$ where $\theta$ is the weights of the neural network used to determine ...
1
vote
0answers
22 views

What is best(or proper) normalization method for A3C?

A3C[1] is an asynchronous online learning algorithm in deep reinforcement learning and it uses multiple workers to collect the independent samples asynchronously. In my best knowledge, the popular ...
2
votes
1answer
57 views

What is intuition behind high variance of Monte Carlo method? [closed]

I'm studying Reinforcement learning from lectures of David silver, where he says that Monte Carlo method is not biased and has very high variance. But I don't understand in which sense the bias and ...
7
votes
2answers
105 views

Is Reinforcement Learning the right choice for painting like Bob Ross?

My workplace is having a 2-week code challenge that involves producing an algorithm to reproduce 100 sample Bob Ross paintings as closely as possible given some constraints: "Paintings" are submitted ...
2
votes
1answer
50 views

DQN with XGBoost

Normally a DQN, uses a neuronal network to estimate the Q-Value. I have framed my problem as a regression problem before and have observed that XGBoost does outperform a NN. Is it possible to replace ...
0
votes
0answers
16 views

Reinforcement learning: Blind state

I want to have an agent run through a maze, but the agent should be blind. I.e., the agent does not know where he is, only the number of steps he has taken already. My problem now is, when ...
2
votes
1answer
23 views

Reinforcement Learning split in training and test set necessary?

Currently I am learning Reinforcement Learning and I am wondering if it's necessary to split the data in a training and test set? Furthermore, in the examples I have seen the models are trained in ...
1
vote
0answers
22 views

Artificial Intelligence to Navigate websites [closed]

I have a requirement to create a bot that could Login to a website with credentials and then extract specific information from the website. Also the bot should be capable of doing the same on ...
3
votes
1answer
49 views

How do the Atari games constitute a finite MDP?

I am still new to Deep-RL. I was reading DeepMind's paper "Playing Atari with Deep Reinforcement learning" and am having trouble understanding how these games can be represented as a finite markov ...
1
vote
0answers
17 views

Does TRPO train for multiple steps on the same data?

I understand mainly how Trust Region Policy Optimisation (TRPO) works, in terms of the trust region constraining the gradient updates from straying too far from the 'old' policy. Does this mean that ...
1
vote
1answer
20 views

Reinforcement learning, how can I get the entropy of an distribution?

I read a paper Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments, and I found the approximate policy is learned by maximizing the log probability of agent $j$’s actions, with an ...
0
votes
1answer
27 views

How can I calculate the variance of policy gradient method? [closed]

In Reinforcement Learning, how can I calculate the variance of policy gradient method?