A message from our CEO about the future of Stack Overflow and Stack Exchange. Read now.

Questions tagged [q-learning]

A popular reinforcement learning algorithm, an instance of TD (temporal difference) learning.

Filter by
Sorted by
Tagged with
0
votes
0answers
11 views

Expected SARSA, SARSA and Q-learning

I would much appreciate if you could point me in the right direction regarding this question about targets for approximate ...
1
vote
1answer
22 views

Deep Q Learning best practice

I'm new in deep q-learning and I have understood the main concepts of it and I'm trying to solve problems with DQL. The problem is that I don't know how to initialize some key values of the algorithm ...
0
votes
1answer
14 views

Small difference of q-function between different actions for the same state

I am trying out reinforcement learning using Q-learning. The data come from some made-up equations so I have infinite number of data. One thing that troubles me is after I learn the Q-function, I use ...
3
votes
1answer
33 views

DQN - agent doesn't improve policy

I have a simple grid environment. The player is in the upper left corner and it's goal is to get to lower right corner. The player receives +0.2 points for moving in the direction of the goal, -0.2 ...
1
vote
1answer
56 views

Delayed Rewards in Reinforcement Learning

I have an MDP where the rewards are delayed for six steps as follows: The reward from action at time t is received when the action at time t+6 is taken. The reward from action at time t+1 is ...
0
votes
0answers
34 views

How to use data from other policies in order to find optimal policy in model-free rl?

I am struggling to understand whether experience from one policy can be used to find optimal policy. Suppose that I have gathered many data (state, action, reward, next_state) by following random ...
0
votes
0answers
104 views

Q-learning agent stucks in an infinite loop

I am simulating a mouse to find a cheese on an empty table. I randomly put a cheese on the table and let the mouse find the cheese without falling off the table. The problem is, in test part, agent ...
2
votes
1answer
29 views

Rationale behind Q-learning

I am reading Sutton Barto on Reinforcement Learning. I understand that $TD(\lambda)$ methods propose better performance than Monte Carlo methods, with TD methods combining advantages of Dynamic ...
1
vote
1answer
89 views

Reward attribution in deep q learning and texas holdem poker

I’m having issues with reward attribution in poker using deep q learning. Multiple actions will yield one reward, but the reward is only known at the end of the hand, not before. I have built a gym ...
1
vote
1answer
24 views

Why not sample action from Q values?

When collecting experience from which to estimate a Q(s,a) function, one common technique in the literature is to follow an epsilon greedy-strategy. In this strategy, the agent selects a random action ...
0
votes
1answer
81 views

How does the Dyna Q algorithm works?

I'm having a hard time trying to understand how the dyna Q algorithm works. I put the picture which helps me to understand. My questions are: What planning really means? (it's the (f) in this picture)...
0
votes
0answers
219 views

Episodic Semi-gradient Q-learning for Estimating approximation of optimal action-value function

at page 244 of Sutton and Barto book on Reinforcement Learning (book) is described the pseudocode for episodic semi-gradient Sarsa, while it is never given a pseudocode for the corresponding episodic ...
1
vote
1answer
129 views

DQN - How to feed the input of 4 still frames from a game as one single state input

I was reading this blog about Deep Q-Learning. 1- In the The input section of the blog, I wanted to know how do we feed the 4 still-frames/screenshots from the game, that represent the input state, ...
0
votes
1answer
62 views

DQN - breaking correlation between consecutive samples and random sampling

I was reading through some blogs about Deep Q-Learning (DQN), and I have 2 questions: 1- I didn't understand how breaking the correlation between consecutive samples (i.e. train the network with ...
0
votes
0answers
24 views

Q-learning python implementation

I have tried to recreate the model in this post. Essentially, the agent (denoted player) starts at the beginning of a 1x5 matrix and can move either forward or backward (but only foward if at position ...
0
votes
0answers
46 views

Why is the reward fluctuating for Double Q-Learning?

I am trying to implement Double Q-Learning using neural networks from the Keras library. When I first tried Simple DQN, the graph of the reward was fluctuating a lot so, I implemented a Double DQN. ...
0
votes
0answers
25 views

Is there a branch of machine learning that can deal with near infinite state spaces

So I have a game type problem defined as follows; Up to 10 players Each player has: 64 tiles 200 piece types Up to 20 pieces in play at any time There's a random ...
0
votes
1answer
31 views

How do Tile Coding offsets still cover full state space / affect edge cases?

Reading Sutton & Barto I’m having a hard time visualizing the implementation of the tile coding discretization of states. Specifically, if tilings are offset, how does this effect edge cases? For ...
0
votes
1answer
34 views

Different algorithms categorized in reinforcement learning

For some time I am going through reinforcement learning, and have found a lot of diverse information specially in area of Policies (algorithms). I figured out that policies can be classified in On ...
0
votes
0answers
19 views

Reward function for parking vehicle

I am having problems with parking a vehicle using DQN and could use some help. Agent Description The agent exists in a square grid with boundaries (X x Y), and is only able to move forward, turn ...
1
vote
1answer
224 views

Differences between Sarsa and Q-learning control procedural algorithms

I am referring to pages 130-131 of Sutton and Barto book on Reinforcement Learning available here: book I don't understand the slight difference that there is between the two procedural algorithms ...
0
votes
1answer
255 views

Limits and constraints for Q-learning

I have simple implementation of Q-learning algorithm and I'm trying to run it on States space size = 36865 Actions space size = 25 So my resulting Q-table is ...
1
vote
1answer
116 views

Q-learning shows worse results than value iteration

I'm trying to solve the same problem with different algorithms (Travel max possible distance with a car). While using value iteration and policy iteration I was able to get the best results possible ...
0
votes
1answer
28 views

Reinforcement learning based Q-learning for wireless routing

In the Q-learning method to get the optimal strategy, the update method is like the following: \begin{equation} Q(S,A) \leftarrow \ Q(S,A) + \alpha [R+\gamma~max_a(Q(s',a)) -Q(S,A)] \end{equation} If ...
1
vote
0answers
29 views

Vanishing reward function in Q-Learning

Imagine that the agent receives a positive reward upon reaching a state $s$. Once the state $s$ has been reached the positive reward associated with it vanishes and appears somewhere else in the state ...
0
votes
1answer
377 views

How to explain and visualize a Q Learning Agent?

What are some common visualization approaches used in explaining the behavior of a Q-Learning agent? Here is an excerpt of some example Q values for 5 actions serialized to json: ...
0
votes
1answer
69 views

How does DQN parameter updates work in simulation?

I've already read almost every Questions-answers and material related to DQN, deep reinforcement learning, but I'm struggling to start working on simulation. First of all, I'm trying to code using ...
2
votes
0answers
46 views

Deep Q - Learning Exploration - BestQ Value

I am trying to implement a Deep Q - Network to play Asteroids. Unfortunately, I am not sure how to calculate the Q Value exactly if I am exploring. For example, the Agent is exploring for 1 second (...
1
vote
0answers
116 views

Initialize replay memory and action value function Q

I am not sure I can ask this question here, but I will make an attempt. I am trying to implement Beat Atari with Deep Reinforcement Learning. They explained all steps very well, but they ask you to ...
0
votes
0answers
98 views

How to apply Reinforcement Learning to time-series sampling?

I want to apply the concept of Reinforcement Learning (RL) to help an agent decide when to sample an unobserved time-series signal. Can you help/guide me on how to proceed? The problem setup can be ...
0
votes
1answer
137 views

How to apply multi agent deep reinforcement learning to an environment with discrete action space

Do you know or have heard about any cutting edge deep reinforcement-learning algorithm which can be successfully applied for discrete action-spaces in multi-agent settings? I have been researching ...
1
vote
0answers
25 views

Convergence criterion for R-learning algorithm

I'm trying to find a policy for a simple game using R-learning algorithm. I have a field with values (agent can move in 4 directions) and the goal is to get from starting point to finish point with ...
4
votes
2answers
430 views

Value iteration does not converge when using Q learning

I have a simple game and want my agent to play it with a help of reinforcement learning. We have a board and a value in each cell. The goal is to go from start to finish point with the highest score (...
2
votes
2answers
119 views

Why in Q-Learning, policy pi is evaluated through another policy u?

I've been watching David Silver's courses about Reinforcement Learning. According to his lectures, policy $\pi$ is evaluated by evaluating another policy $\mu$. But I cannot understand: why is it so? ...
0
votes
0answers
52 views

Adding additional constrains to OpenAi Gym

I'm currently working trough some examples which should finally end in a DQN Reinforcement Learning for the CartPole example in the openAI-Gym. Copied some code from GitHub which isn't deep yet: <...
5
votes
1answer
97 views

Understanding Q-learning for continuous actions

I am reading the paper on Normalized Advantage Functions for continuous Q-learning and I am having trouble understanding why the advantage function takes this particular form: Why is the Advantage ...
3
votes
1answer
258 views

Why is this the correct formula to update the NN weights in Q-learning?

I'm trying to implement Q-learning to train an AI bot to play Pokemon battles. Since there is a large state space (corresponding to all possible states a battle can have in between moves), I can't use ...
1
vote
0answers
541 views

SARSA with Linear Function Approximation weight overflow

I'm trying to solve the CartPole problem, implemented in OpenAI Gym. In each state the agent is able to perform one of 2 actions move left or right. The reward is always +1. The epsiode ends after 500 ...
0
votes
0answers
22 views

Neural network how to deal with comparison

I'm currently working on a DQN network and this question comes to me. As far as I know, neural networks are good at dealing with values that have never seen (generalisation). E.g. If a classification ...
1
vote
2answers
648 views

Why update SARSA with S'A' at all if the goal is a less aggressive exploitation policy?

Why is it that we update the Q values using S' and A' and not the maximum as in Q-learning? If the goal is to have a less aggressive exploitation policy, why don't we simply use an epsilon greedy ...
0
votes
2answers
41 views

Q-learning with 100-500 discrete actions

I've a Reinforcement Learning problem where I want to learn the Q function. For action space of size in the order of 100s is Q learning a good option? Will it converge?
1
vote
1answer
111 views

What is momentum-like training aid technique for reinforcement learning (Q learning)?

Is there any method that could help a reinforcement learning (specifically Q learning) model converge? Can optimization strategy like momentum/Adam/RMSProp applied to Q learning to update Q-table ...
1
vote
1answer
164 views

Q learning: overtraining and converagence

I'm working on a Q learning model to autopilot Flappy Bird (follow http://sarvagyavaish.github.io/FlappyBirdRL/): it manage to reach a good score like 500 after a while of training: But after longer ...
0
votes
1answer
113 views

What could be the causes of weights not changing during backpropagation?

I'm currently working on a Dueling-Double DQN model, and I noticed that though the loss (mse of Q values between training and target networks) seems to be decreasing, the distribution of weights in ...
2
votes
2answers
93 views

Deep Q-Learning: Experience replay overriding old Memories?

This is my first question on SE in general. So if I make any mistakes - please feel free to point them out to me. My Question is about Deep Q-Learning. I've been working into some code examples and ...
3
votes
1answer
151 views

Reward function for grid based path planning deep Q-learning agent

I'm really getting stuck on creating a good reward function for my agent and could use some advice. I'll explain the setting for my question first: Agent Description The agent in question exists in ...
7
votes
1answer
506 views

Is Deep-Q Learning inherently unstable

I'm reading Barto and Sutton's Reinforcement Learning and in it (chapter 11) they present the "deadly triad": Function approximation Bootstrapping Off-policy training And they state that an ...
0
votes
1answer
172 views

Is double Q-learning redundant when using target networks?

Generally speaking, the purpose behind target networks is to reduce the impact of current changes on the model. i.e. if I performed action a and got some reward <...
1
vote
1answer
79 views

How can Deep Q Learning be applied to scenarios with rewards only received in a final step?

I am applying DQ Learning to a continuous action space with rewards received at the end of each trial. My agent is in a fixed 24step long setting where it receives the reward at the end of those 24 ...
1
vote
1answer
156 views

Experience replay, why store SARS' and not SAQRS'

For Q-learning Experience replay, why do we store into the bank observations: { stateFrom, actionIx, imediateReward, resultingState } instead of { stateFrom, actionIx, actionQValue, ...