Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange
Make your voice heard. Take the 2019 Developer Survey now

Questions tagged [q-learning]

A popular reinforcement learning algorithm, an instance of TD (temporal difference) learning.

0
votes
1answer
12 views

How does DQN parameter updates work in simulation?

I've already read almost every Questions-answers and material related to DQN, deep reinforcement learning, but I'm struggling to start working on simulation. First of all, I'm trying to code using ...
2
votes
0answers
25 views

Deep Q - Learning Exploration - BestQ Value

I am trying to implement a Deep Q - Network to play Asteroids. Unfortunately, I am not sure how to calculate the Q Value exactly if I am exploring. For example, the Agent is exploring for 1 second (...
1
vote
0answers
22 views

Initialize replay memory and action value function Q

I am not sure I can ask that question here, but I will try an attempt. I am trying to implement Beat Atari with Deep Reinforcement Learning. They explained very well each steps, but they ask you to ...
0
votes
0answers
21 views

Why isn't my reinforcement learning agent learning anything useful?

I've been trying to implement a Q-learning agent to play the game of snake. There are many examples of deep Q-learning agents doing this on github but I couldn't find any with simple Q-learning and as ...
0
votes
0answers
54 views

How to apply Reinforcement Learning to time-series sampling?

I want to apply the concept of Reinforcement Learning (RL) to help an agent decide when to sample an unobserved time-series signal. Can you help/guide me on how to proceed? The problem setup can be ...
0
votes
1answer
36 views

How to apply multi agent deep reinforcement learning to an environment with discrete action space

Do you know or have heard about any cutting edge deep reinforcement-learning algorithm which can be successfully applied for discrete action-spaces in multi-agent settings? I have been researching ...
1
vote
0answers
14 views

Convergence criterion for R-learning algorithm

I'm trying to find a policy for a simple game using R-learning algorithm. I have a field with values (agent can move in 4 directions) and the goal is to get from starting point to finish point with ...
4
votes
2answers
92 views

Value iteration does not converge when using Q learning

I have a simple game and want my agent to play it with a help of reinforcement learning. We have a board and a value in each cell. The goal is to go from start to finish point with the highest score (...
2
votes
2answers
48 views

Why in Q-Learning, policy pi is evaluated through another policy u?

I've been watching David Silver's courses about Reinforcement Learning. According to his lectures, policy $\pi$ is evaluated by evaluating another policy $\mu$. But I cannot understand: why is it so? ...
0
votes
0answers
41 views

Adding additional constrains to OpenAi Gym

I'm currently working trough some examples which should finally end in a DQN Reinforcement Learning for the CartPole example in the openAI-Gym. Copied some code from GitHub which isn't deep yet: <...
5
votes
1answer
42 views

Understanding Q-learning for continuous actions

I am reading the paper on Normalized Advantage Functions for continuous Q-learning and I am having trouble understanding why the advantage function takes this particular form: Why is the Advantage ...
3
votes
1answer
143 views

Why is this the correct formula to update the NN weights in Q-learning?

I'm trying to implement Q-learning to train an AI bot to play Pokemon battles. Since there is a large state space (corresponding to all possible states a battle can have in between moves), I can't use ...
1
vote
0answers
198 views

SARSA with Linear Function Approximation weight overflow

I'm trying to solve the CartPole problem, implemented in OpenAI Gym. In each state the agent is able to perform one of 2 actions move left or right. The reward is always +1. The epsiode ends after 500 ...
0
votes
0answers
21 views

Neural network how to deal with comparison

I'm currently working on a DQN network and this question comes to me. As far as I know, neural networks are good at dealing with values that have never seen (generalisation). E.g. If a classification ...
1
vote
2answers
239 views

Why update SARSA with S'A' at all if the goal is a less aggressive exploitation policy?

Why is it that we update the Q values using S' and A' and not the maximum as in Q-learning? If the goal is to have a less aggressive exploitation policy, why don't we simply use an epsilon greedy ...
0
votes
2answers
31 views

Q-learning with 100-500 discrete actions

I've a Reinforcement Learning problem where I want to learn the Q function. For action space of size in the order of 100s is Q learning a good option? Will it converge?
1
vote
1answer
43 views

What is momentum-like training aid technique for reinforcement learning (Q learning)?

Is there any method that could help a reinforcement learning (specifically Q learning) model converge? Can optimization strategy like momentum/Adam/RMSProp applied to Q learning to update Q-table ...
1
vote
1answer
80 views

Q learning: overtraining and converagence

I'm working on a Q learning model to autopilot Flappy Bird (follow http://sarvagyavaish.github.io/FlappyBirdRL/): it manage to reach a good score like 500 after a while of training: But after longer ...
0
votes
1answer
43 views

What could be the causes of weights not changing during backpropagation?

I'm currently working on a Dueling-Double DQN model, and I noticed that though the loss (mse of Q values between training and target networks) seems to be decreasing, the distribution of weights in ...
0
votes
0answers
10 views

Features Observability In Sequential Data For Deep Network

I have an unconventional classification problem over sequential data (at least to me since I am relatively new to this area). The problem goes as follows. For time index $i$ we observe features $F_i$ ...
1
vote
2answers
53 views

Deep Q-Learning: Experience replay overriding old Memories?

This is my first question on SE in general. So if I make any mistakes - please feel free to point them out to me. My Question is about Deep Q-Learning. I've been working into some code examples and ...
3
votes
1answer
56 views

Reward function for grid based path planning deep Q-learning agent

I'm really getting stuck on creating a good reward function for my agent and could use some advice. I'll explain the setting for my question first: Agent Description The agent in question exists in ...
5
votes
1answer
205 views

Is Deep-Q Learning inherently unstable

I'm reading Barto and Sutton's Reinforcement Learning and in it (chapter 11) they present the "deadly triad": Function approximation Bootstrapping Off-policy training And they state that an ...
0
votes
1answer
106 views

Is double Q-learning redundant when using target networks?

Generally speaking, the purpose behind target networks is to reduce the impact of current changes on the model. i.e. if I performed action a and got some reward <...
1
vote
1answer
56 views

How can Deep Q Learning be applied to scenarios with rewards only received in a final step?

I am applying DQ Learning to a continuous action space with rewards received at the end of each trial. My agent is in a fixed 24step long setting where it receives the reward at the end of those 24 ...
1
vote
1answer
89 views

Experience replay, why store SARS' and not SAQRS'

For Q-learning Experience replay, why do we store into the bank observations: { stateFrom, actionIx, imediateReward, resultingState } instead of { stateFrom, actionIx, actionQValue, ...
1
vote
1answer
83 views

Why semi-gradient is used instead of the true gradient in Q-learning?

I am asking a duplicated question that nobody has answered yet. In Q-learning with function approximation, the objective is to minimize MSE between the target $r + \gamma \max_{a'} Q(s',a',w)$ and the ...
0
votes
1answer
20 views

Q-learning with slow rewards [closed]

A team in my company has implemented a basic model-free Q-Learning agent in relation to inventory control. The problem (in my eyes) is that it only knows its reward once per day based on revenue gain ...
17
votes
2answers
3k views

Why was the letter Q chosen in Q-learning?

Why the letter Q was chosen in the name of Q-learning? Most letters are chosen as an abbreviation, such as $\pi$ standing for policy and $v$ stands for value. But I don't think Q is an abbreviation ...
1
vote
1answer
61 views

Q-Learning: state independent of agent's action [closed]

Could state be independent of the action chosen by agent? We would have a situation in which agent learns only which actions are the best in specific states without having any impact on those states (...
0
votes
0answers
213 views

Is Monte Carlo Tree Search policy or value iteration (or something else)?

I am taking a Reinforcement Learning class and I didn’t understand how to combine the concepts of policy iteration/value iteration with Monte Carlo (and also TD/SARSA/Q-learning). In the table below, ...
0
votes
1answer
23 views

Optimality in Hierarchy of Machines (HAM Framework)

How is it that the HAM framework provides the Hierarchically Optimal solution while the MAXQ framework provides the recursively optimal solution? The above statement is based on Section ...
2
votes
0answers
11 views

Why the invariant reward helps training?

I am new to Machine Learning, and I am trying to solve MountainCar-v0 using Q-learning. I can solve the problem now, but I am still confused. According to the MountainCar-v0's Wiki, the reward ...
1
vote
1answer
41 views

In MDPs with deterministic actions, should I use Q-learning or TD(0)?

Suppose in an Markov Decision Process (MDP), we have transition $(s, a, r, s', a', r', s'', ...)$, learning rate $\alpha$ and discount factor $\lambda$. The update formula of $TD(0)$: $V(s) \...
0
votes
1answer
62 views

Why Deep Qlearning is better than Qlearning? [closed]

If Q-learning is supposed to converge toward the optimal policy, how is it possible to do better?
2
votes
1answer
205 views

Does higher maximum Q value imply better or worse performance?

Suppose I run a RL algorithm and for every episode, I grab the (average) maximum Q value. I do this for several runs (with different hyper-paramters, for instance) to compare performance of my RL ...
1
vote
2answers
501 views

How to perform deep Q-learning batch update step on a neural network with multiple outputs

I am taking on deep Q-learning and I am stuck at understanding one particular thing. I have googled multiple deep Q-learning examples, but literally everyone posting tutorials uses a cart-pole game to ...
5
votes
3answers
900 views

Why don't we use importance sampling for one step Q-learning?

Why don't we use importance sampling for 1-step Q-learning? Q-learning is off-policy which means that we generate samples with a different policy than we try to optimize. Thus it should be ...
0
votes
1answer
119 views

How can I minimize future long-term reward in deep Q-learning? [closed]

I’m trying to implement deep Q-learning on a problem were the rewards the agent receives are errors from another model. The RL agents job is to minimize the long-term reward (error) instead of ...
2
votes
1answer
229 views

Confused about Function Approximation for Q Learning

I am not sure that I understood Funtion approximation for Q Learning. So basicall with FA we don't use tables anymore? Each state is now represented with features, and we multiply those features with ...
5
votes
2answers
464 views

Is planning in Dyna-Q a form of experience replay?

In Richard Sutton's book on RL (2nd edition), he presents the Dyna-Q algorithm, which combines planning and learning. In the planning part of the algorithm, the Dyna-agent randomly samples n state-...
0
votes
0answers
119 views

Reinforcement learning: multiplayer game state vector for variable number opponents

As far as I know, in many of the recent deep reinforcement learning papers, such as DQN, etc., it seems like the state vector parameterization always has the same dimension. Sure, they often do some ...
0
votes
1answer
201 views

Parameterizing Reinforcement Learning card game state space

I want to model a particular card game as a reinforcement learning problem. For simplicity let's say it is a single standard 52 card deck, and let's say it is just 2 players. The exact details are not ...
1
vote
0answers
26 views

Possible Learning Approach

I am working on a reinforcement learning problem where I have 8 different actions [1 2 3 4 5 6 7 8] and a symmetric state matrix of 10 by 10 (only upper or triangular matrix is sufficient). I know I ...
9
votes
1answer
1k views

Overview over Reinforcement Learning Algorithms

I'm currently searching for an Overview over Reinforcement Learning Algorithms and maybe a classification of them. But next to Sarsa and Q-Learning + Deep Q-Learning I can't really find any popular ...
6
votes
1answer
939 views

Q-learning when to stop training?

I'm using Q-learning for my side project. After few million episodes, I found the cumulative rewards seems to reach stable. I'm wondering if there's a scientific way(s) to determine when to stop ...
1
vote
1answer
316 views

What are the differences between contextual bandits, actor-citric methods, and continuous reinforcement learning?

Let's imagine we have a blackbox function f(X) -> y which we don't know. X is a vector of 10 continuous variables, which we ...
4
votes
2answers
502 views

SARSA update rule

The update rule for the SARSA algorithm, which is mentioned here is the following. $Q(s_t,a_t) \leftarrow Q(s_t,a_t) + \alpha [r_{t+1} + \gamma Q(s_{t+1}, a_{t+1})-Q(s_t,a_t)]$ My question is, why ...
0
votes
1answer
286 views

What happens in the fully connected layer of a Deep-Q-Network?

I am currently reading Deepminds "Playing Atari with Deep Reinforcement Learning"-Paper (2013) and since I don't quite understand it, I would like to ask here. What exactly happens in each fully ...
1
vote
1answer
272 views

Why do networks used for Deep Q-Learning have several outputs?

I try to implement a deep Q-Learning solution for the the Openai gym CartPole problem. My solution doesn't work as my network takes forever to learn anything useful. I compared my solution with other ...