Questions tagged [q-learning]
A popular reinforcement learning algorithm, an instance of TD (temporal difference) learning.
163
questions
0
votes
1
answer
20
views
Several Questions About "Prioritized Experience Replay"
I have 3 questions about "Prioritized Experience Buffer" as described in the paper.
what's the point of the importance sampling (IS)?
I'll explain - I understand that:
a. when we ...
- 223
0
votes
0
answers
10
views
In a convolutional layer, is it standard practice to modify stride and padding to get a desired output?
I'm trying to implement the CNN described in A Framework of Hierarchical Deep Q-Network for Portfolio Management (see screenshot). In the paper, the author describes the first CNN layer as having a ...
0
votes
0
answers
31
views
How is Q learning applied to a DQN?
I'm a beginner when it comes to Reinforcement Learning and ML in general and I'm looking into Q Learning with respect to the Cart pole problem. Below is some code taken from an implementation I found
(...
1
vote
1
answer
233
views
What is the difference between an Epoch and an Episode?
I am working on DQN and have confused myself with Epoch and Episode. I have gone through this answer but the confusion is increased.
I will explain my scenario and would like to understand the ...
- 113
1
vote
0
answers
102
views
What is the maximum Q-value that any state can obtain in DQN?
The q value of a specific state,$s$ and action, $a$ is given by the following equation, as per Sutton and Barto's equation 3.13 -
$$q_{\pi}(s,a) = \mathbb{E}_{\pi}[\sum_{k=0}^{\infty}\gamma^{k}R_{t+k+...
- 448
0
votes
0
answers
14
views
Expected number of random cell visits in Q-learning
I'm interested in Q-learning with a time-decaying exploration rate
$$ \varepsilon_t = \exp(-\beta t)$$
In order to evaluate how `reasonable' certain values of $\beta$ are, and to make them comparable ...
- 1,045
1
vote
0
answers
21
views
DQN network cannot distinguish output ports with different actions
I am trying to implement a Deep Q-Network (DQN) with a prediction network and a target network. Both nets take the state as input and have two output ports each corresponding to a different action. ...
- 11
1
vote
1
answer
74
views
How does Deep Reinforcement Learning remove the need to map or explore every state, action pair for an agent?
I am interested in using Deep Reinforcement Learning to teach an AI how to play a game, where the AI knows the model of the game at the start (so I would use model-based deep reinforcement learning?)
...
Awepossum
3
votes
2
answers
136
views
Can reinforcement learning directly work with advantage function?
When we are at a state s, we only need to determine the relative performance among different actions in order to choose the optimal action. In other words, we only ...
- 1,823
2
votes
1
answer
133
views
Reinforcement Learning with Oracle Policy
I'm working on a reinforcement learning problem. The simulation environment is pretty simple (like those maze problems) so I can manually work out its solution. The idea I have is: since I can work ...
- 1,823
1
vote
0
answers
115
views
In deep Q-learning how is the error calculated for each output node?
In the Q-learning approach to reinforcement learning where a neural network learns to approximate the Q-function, I believe that the output layer has one node for each possible action ($a_i$) and the ...
- 63
2
votes
1
answer
157
views
Simultaneous actions with conditional legality for reinforcement learning agent
I'm training a reinforcement model playing a game with self-learning. For each state, the agent can select one or several simultaneous actions from a list of possible actions. One possible action is a ...
- 153
3
votes
1
answer
360
views
Setting hyper-parameters for Deep Q-learning
I would like to seek views on the appropriate hyper-parameters for an implementation of Deep Q-learning. Thanks for your help.
I’m following the implementation from the 2015 paper Deep Reinforcement ...
- 31
3
votes
1
answer
115
views
Bellman Optimality Operator fixed point
I'm reading Szepesvári's book on RL. My question is concerning the proof of Theorem A.10 (p. 71).
Theorem
Let $V$ be the fixed point of $T^∗$ and assume that there is policy $π$ which is greedy w.r.t ...
- 43
0
votes
1
answer
51
views
How is the Q value translated to action in DRL framework presented in Gu, Shixiang et al (2016)
I was reading the paper Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updates and I kind of understand everything. The results also seem very interesting, enabling ...
2
votes
1
answer
592
views
Policy improvement in SARSA and Q learning
I have a rather trivial doubt in SARSA and Q learning. Looking at the pseudocode of the two algorithms in Sutton&Barto book, I see the policy improvement step is missing.
How will I get the ...
- 391
1
vote
0
answers
460
views
Q-learning convergence with stochastic reward function
Every proof for convergence of Q-learning I can find assumes that the reward is a function $r(s, a, s')$ i.e. deterministic. However, MDPs are often defined with a stochastic reward, as exemplified in ...
- 131
2
votes
1
answer
465
views
Why Does Double-Q Learning Converge So Slowly?
Working through Sutton and Barto's reinforcement learning book and van Hasselt's original Double Q-Learning (2010) paper to understand Double-Q learning. In the original paper, van Hasselt uses the ...
- 167
0
votes
1
answer
225
views
Deduce the Bellman equation from the Value and Q functions
I am trying to derive/deduce the bellman equation using Value and Q-functions.
I came only so far with understanding it and tried it myself in Latex:
Why is the $V^*$ suddenly in $Q^\pi$ function? ...
0
votes
1
answer
62
views
q-learning for classification problem
how can I use q-learning for classification problem I see some examples about learning the agent in a game I need to use q-learning for binary classification (classify image if it is for cat or for ...
1
vote
0
answers
44
views
ε-Greedy with Q learning / SARSA can have stochastic policy?
Hello I'm now studying Q learning and SARSA with ε-Greedy , Softmax startegies. And have a question about my readings.
In my readings,
when SARSA with ε-Greedy, SARSA causes value-function ...
- 11
1
vote
1
answer
195
views
Why does the PyTorch tutorial on DQN define state as a difference?
I'm a master's student in EECS working my way towards understanding how DQN [0] works.
I'm working towards solving the CartPole-v0 task in as few iterations as possible.
First of all I implemented a ...
- 70
7
votes
1
answer
191
views
MDP and Sate Value Finding?
I have a complex MDP (I think) as follows. anyone can describe me simply how the value for state $V(A)^*$ is find?
First Update: really for this solved question I need a canonical answer, step by ...
3
votes
1
answer
356
views
SARSA when policy is not epsilon greedy
I would like to clarify a doubt that I have regarding SARSA. SARSA can be used for optimal control when the policy to take action $a$ is epsilon greedy. Suppose that the policy to take action $a$ is ...
- 1,066
3
votes
1
answer
302
views
Association between Current state/action and the far future reward
How the Agent make the association between the current $Q_t(s_t,a_t)$ and a far future reward that by nature of my environment we get reward at least after 10-15 time steps from the action taken. If ...
- 73
0
votes
1
answer
505
views
Understanding the Q-learning loss function?
Perhaps this can be explained a little more to me. I understand what's in literature but I'm struggling to understand why this is the preferred loss.
If we have an agent that can move ↑↓→← and for ...
3
votes
1
answer
208
views
Learning in the bit flipping environment
I'm looking at the Andrychowicz et al. paper and running through the gauntlet of implementing a DQN and then implementing a DQN with HER.
In this paper, they mention a bit-flipping environment: a ...
- 133
0
votes
0
answers
197
views
As epsilon decays, rewards gets worse during exploitation than exploration
I am currently trying to write learning agent from the "Human Level Control in DRL" Paper in Tensorflow 2.0. I've copied the recommended hyperparameters and picked the easiest environment ...
- 71
1
vote
0
answers
247
views
Why and under what conditions does Q learning converge?
I am looking for a modern proof on why Q learning converges in the tabular setting.
I've skimmed the original proof by Dayan and Watkins and I have to say that the terminology and approach are a bit ...
- 343
2
votes
1
answer
241
views
Does gradient descent work for tabular Q learning?
Suppose I have a tabular Q learning problem such as grid-world.
Let our loss be defined as,
$$\hat{L}(Q)=0.5(Q(s,a)-(r+\gamma\max_{a'}{Q(s',a')}))^2$$
Then $Q_{k+1}(s,a) = Q_k(s,a) - \eta \nabla \hat {...
- 317
1
vote
2
answers
256
views
In Reinforcement Learning (DQN), is there a way to constrain/penalise the model so that it doesn't take a different action very often?
The RL model I am building is one form of DQN. Its internal network is a regular deep NN.
In the application I am looking at, there is a cost for taking a different action (compared to the previous ...
- 31
1
vote
0
answers
131
views
How to define a Reinforcement Learning episode when dealing with dataset of customer's features?
so one of friends sent me some problem he was working on lately and I couldn't help but to wonder how could it be solved using Q-learning. The statement is as follows :
Given the following datasets, ...
- 11
2
votes
0
answers
49
views
Preprocessing data for the learning step
I am currently reading "Human level control through deep reinforcement learning" and I came across the algorithm in the paper. I am confused because the algorithm uses a different notation ...
- 71
-1
votes
1
answer
151
views
Deep Q learning algorithm
Ok I’m a bit confused with this code, what exactly is a time step, isn’t it like when an action is performed,it goes to the next time step, and also, the gradient descent steps is a repeat until ...
- 1
2
votes
1
answer
288
views
In a DQN, can Prioritized Experience Replay actually perform worse than a regular Experience Replay?
I've written a Double DQN-based stock trading bot using mainly time series stock data.
I've recently upgraded my Experience Replay(ER) code with a version of Prioritized Experience Replay (PER) ...
- 31
3
votes
2
answers
567
views
Can an agent be trained in a completely random environment (the rules and actions stay the same)
I am specifically talking about the FrozenLake example from openAI gym to illustrate my question.
https://gym.openai.com/envs/FrozenLake-v0/
The way I understand how Q-Learning or DQL works on the ...
- 213
0
votes
1
answer
151
views
Consider more than just a single state-action pair before giving a reward in deep q learning?
TLDR If the environment is rather complex (momentum, velocity, ...), but the action space is rather easy (left, right), is it not better to take multiple decisions into account before giving a reward ...
- 213
3
votes
1
answer
191
views
Q-learning update formula
I am a beginner and it is my first question.
I know that Q-learning update equation is:
$Q(s_t, a_t) = Q(s_t, a_t)+α(r_{t+1} +γ·max_AQ(s_{t+1}, a_t)−Q(s_t, a_t))$
But in some of the researches it is ...
- 131
0
votes
1
answer
82
views
Q-Learning [Sutton]: why random variable in formula
Sutton et al. use throughout their book Reinforcement Learning capital letters to describe random variables. At page 131 they introduce Q-Learning.
$Q(S_t,A_t)\leftarrow Q(S_t,A_t) + \alpha [R_{t+1} ...
- 3
1
vote
0
answers
312
views
Expected SARSA, SARSA and Q-learning
I would much appreciate if you could point me in the right direction regarding this question about targets for approximate ...
- 111
1
vote
1
answer
858
views
Deep Q Learning best practice
I'm new in deep q-learning and I have understood its main concepts and I'm trying to solve problems with DQL. The problem is that I don't know how to initialize some key values (AKA hyperparameters) ...
- 119
1
vote
1
answer
32
views
Small difference of q-function between different actions for the same state
I am trying out reinforcement learning using Q-learning. The data come from some made-up equations so I have infinite number of data.
One thing that troubles me is after I learn the Q-function, I use ...
- 1,823
3
votes
1
answer
778
views
DQN - agent doesn't improve policy
I have a simple grid environment. The player is in the upper left corner and it's goal is to get to lower right corner. The player receives +0.2 points for moving in the direction of the goal, -0.2 ...
- 131
2
votes
1
answer
1k
views
Delayed Rewards in Reinforcement Learning
I have an MDP where the rewards are delayed for six steps as follows:
The reward from action at time t is received when the action at time t+6 is taken.
The reward from action at time t+1 is ...
- 691
0
votes
0
answers
49
views
How to use data from other policies in order to find optimal policy in model-free rl?
I am struggling to understand whether experience from one policy can be used to find optimal policy.
Suppose that I have gathered many data (state, action, reward, next_state) by following random ...
- 691
1
vote
1
answer
2k
views
Q-learning agent stucks in an infinite loop
I am simulating a mouse to find a cheese on an empty table. I randomly put a cheese on the table and let the mouse find the cheese without falling off the table. The problem is, in test part, agent ...
- 435
3
votes
1
answer
73
views
Rationale behind Q-learning
I am reading Sutton Barto on Reinforcement Learning. I understand that $TD(\lambda)$ methods propose better performance than Monte Carlo methods, with TD methods combining advantages of Dynamic ...
- 623
3
votes
1
answer
598
views
Reward attribution in deep q learning and texas holdem poker
I’m having issues with reward attribution in poker using deep q learning. Multiple actions will yield one reward, but the reward is only known at the end of the hand, not before.
I have built a gym ...
- 301
1
vote
1
answer
102
views
Why not sample action from Q values?
When collecting experience from which to estimate a Q(s,a) function, one common technique in the literature is to follow an epsilon greedy-strategy. In this strategy, the agent selects a random action ...
- 145
1
vote
1
answer
1k
views
How does the Dyna Q algorithm works?
I'm having a hard time trying to understand how the dyna Q algorithm works. I put the picture which helps me to understand. My questions are:
What planning really means? (it's the (f) in this picture)...
- 143