Questions tagged [q-learning]

A popular reinforcement learning algorithm, an instance of TD (temporal difference) learning.

Filter by
Sorted by
Tagged with
0 votes
0 answers
24 views

Can I restrict action space based on state in DDPG?

I am working on a RL based recommender system using the DDPG algorithm and wanted to restrict the action space (possible recommedations) based in the user. I have tried using the restricted action ...
user avatar
0 votes
0 answers
16 views

Variance Minimization with Q-Learning

The problem is actually very simple. There is a sequence of numbers given [10 11 12 13 14]. We will add each of them other numbers between 0 and 9 to minimize the variance of the sequence. As you may ...
user avatar
1 vote
1 answer
31 views

How does Deep Reinforcement Learning remove the need to map or explore every state, action pair for an agent?

I am interested in using Deep Reinforcement Learning to teach an AI how to play a game, where the AI knows the model of the game at the start (so I would use model-based deep reinforcement learning?) ...
user avatar
0 votes
0 answers
20 views

Dyna-Q blocking maze expected future value

Sutton & Barto (2018), Figure 8.4, shows the Dyna-Q algorithm in an environment where the shortest path is blocked at time step 1000: The maximum action values for this problem at the end of the ...
user avatar
  • 1
0 votes
0 answers
8 views

Reinforcement DQ Learning training approach for simple two player game

I have a simple test game where there are two players and they each take turns to play cards from their hand onto the table. They each get three turns, and each card has a word on it (for example &...
user avatar
  • 113
0 votes
0 answers
21 views

Building a simulator for continuous state, discrete action reinforcement learning

I am trying to build a simulator that optimizes the performance and temperature of a device. I want the device to perform well, but without making the device too hot. If the device becomes too hot, I ...
user avatar
3 votes
2 answers
58 views

Can reinforcement learning directly work with advantage function?

When we are at a state s, we only need to determine the relative performance among different actions in order to choose the optimal action. In other words, we only ...
user avatar
  • 1,643
2 votes
1 answer
60 views

Reinforcement Learning with Oracle Policy

I'm working on a reinforcement learning problem. The simulation environment is pretty simple (like those maze problems) so I can manually work out its solution. The idea I have is: since I can work ...
user avatar
  • 1,643
0 votes
0 answers
16 views

Can I converge a Q-learning algorithm by checking the Q-table of two successive episodes?

I'm trying to solve a MDP problem which has non-terminal states and I can get the solution through value iteration(using enumeration) but I get sub-optimal results using Q-learning. I have read ...
user avatar
1 vote
0 answers
49 views

In deep Q-learning how is the error calculated for each output node?

In the Q-learning approach to reinforcement learning where a neural network learns to approximate the Q-function, I believe that the output layer has one node for each possible action ($a_i$) and the ...
user avatar
0 votes
0 answers
32 views

Algorithmic trading - Discretizing or Standardization of Indicators (state)

I am trying out Q-Learner for trading. I have taken three indicators (Price/SMA, BB% and RSI) and these values are continuous variable. The question is, do I have to discretize these indicators and ...
user avatar
  • 101
2 votes
1 answer
78 views

Simultaneous actions with conditional legality for reinforcement learning agent

I'm training a reinforcement model playing a game with self-learning. For each state, the agent can select one or several simultaneous actions from a list of possible actions. One possible action is a ...
user avatar
  • 153
0 votes
0 answers
22 views

Can RL2 be implemented by Q learning?

I want to know whether the RL2<RL2: Fast Reinforcement Learning via Slow Reinforcement Learning> algorithms could be used in Q learning? My confusion is that the hidden state translated into the ...
user avatar
2 votes
1 answer
55 views

Setting hyper-parameters for Deep Q-learning

I would like to seek views on the appropriate hyper-parameters for an implementation of Deep Q-learning. Thanks for your help. I’m following the implementation from the 2015 paper Deep Reinforcement ...
user avatar
1 vote
0 answers
61 views

Bellman Optimality Operator fixed point

I'm reading Szepesvári's book on RL. My question is concerning the proof of Theorem A.10 (p. 71). Theorem Let $V$ be the fixed point of $T^∗$ and assume that there is policy $π$ which is greedy w.r.t ...
user avatar
0 votes
1 answer
32 views

How is the Q value translated to action in DRL framework presented in Gu, Shixiang et al (2016)

I was reading the paper Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updates and I kind of understand everything. The results also seem very interesting, enabling ...
user avatar
2 votes
1 answer
348 views

Policy improvement in SARSA and Q learning

I have a rather trivial doubt in SARSA and Q learning. Looking at the pseudocode of the two algorithms in Sutton&Barto book, I see the policy improvement step is missing. How will I get the ...
user avatar
  • 391
1 vote
0 answers
228 views

Q-learning convergence with stochastic reward function

Every proof for convergence of Q-learning I can find assumes that the reward is a function $r(s, a, s')$ i.e. deterministic. However, MDPs are often defined with a stochastic reward, as exemplified in ...
user avatar
  • 131
2 votes
1 answer
264 views

Why Does Double-Q Learning Converge So Slowly?

Working through Sutton and Barto's reinforcement learning book and van Hasselt's original Double Q-Learning (2010) paper to understand Double-Q learning. In the original paper, van Hasselt uses the ...
user avatar
  • 167
0 votes
1 answer
102 views

Deduce the Bellman equation from the Value and Q functions

I am trying to derive/deduce the bellman equation using Value and Q-functions. I came only so far with understanding it and tried it myself in Latex: Why is the $V^*$ suddenly in $Q^\pi$ function? ...
user avatar
0 votes
1 answer
43 views

q-learning for classification problem

how can I use q-learning for classification problem I see some examples about learning the agent in a game I need to use q-learning for binary classification (classify image if it is for cat or for ...
user avatar
1 vote
0 answers
31 views

ε-Greedy with Q learning / SARSA can have stochastic policy?

Hello I'm now studying Q learning and SARSA with ε-Greedy , Softmax startegies. And have a question about my readings. In my readings, when SARSA with ε-Greedy, SARSA causes value-function ...
user avatar
  • 11
1 vote
1 answer
122 views

Why does the PyTorch tutorial on DQN define state as a difference?

I'm a master's student in EECS working my way towards understanding how DQN [0] works. I'm working towards solving the CartPole-v0 task in as few iterations as possible. First of all I implemented a ...
user avatar
  • 35
7 votes
1 answer
185 views

MDP and Sate Value Finding?

I have a complex MDP (I think) as follows. anyone can describe me simply how the value for state $V(A)^*$ is find? First Update: really for this solved question I need a canonical answer, step by ...
user avatar
3 votes
1 answer
215 views

SARSA when policy is not epilon greedy

I would like to clarify a doubt that I have regarding SARSA. SARSA can be used for optimal control when the policy to take action $a$ is epsilon greedy. Suppose that the policy to take action $a$ is ...
user avatar
  • 876
3 votes
1 answer
148 views

Association between Current state/action and the far future reward

How the Agent make the association between the current $Q_t(s_t,a_t)$ and a far future reward that by nature of my environment we get reward at least after 10-15 time steps from the action taken. If ...
user avatar
  • 43
0 votes
1 answer
191 views

Understanding the Q-learning loss function?

Perhaps this can be explained a little more to me. I understand what's in literature but I'm struggling to understand why this is the preferred loss. If we have an agent that can move ↑↓→← and for ...
user avatar
3 votes
1 answer
136 views

Learning in the bit flipping environment

I'm looking at the Andrychowicz et al. paper and running through the gauntlet of implementing a DQN and then implementing a DQN with HER. In this paper, they mention a bit-flipping environment: a ...
user avatar
  • 133
0 votes
0 answers
91 views

As epsilon decays, rewards gets worse during exploitation than exploration

I am currently trying to write learning agent from the "Human Level Control in DRL" Paper in Tensorflow 2.0. I've copied the recommended hyperparameters and picked the easiest environment ...
user avatar
  • 71
1 vote
0 answers
175 views

Why and under what conditions does Q learning converge?

I am looking for a modern proof on why Q learning converges in the tabular setting. I've skimmed the original proof by Dayan and Watkins and I have to say that the terminology and approach are a bit ...
user avatar
2 votes
1 answer
119 views

Does gradient descent work for tabular Q learning?

Suppose I have a tabular Q learning problem such as grid-world. Let our loss be defined as, $$\hat{L}(Q)=0.5(Q(s,a)-(r+\gamma\max_{a'}{Q(s',a')}))^2$$ Then $Q_{k+1}(s,a) = Q_k(s,a) - \eta \nabla \hat {...
user avatar
  • 297
1 vote
2 answers
192 views

In Reinforcement Learning (DQN), is there a way to constrain/penalise the model so that it doesn't take a different action very often?

The RL model I am building is one form of DQN. Its internal network is a regular deep NN. In the application I am looking at, there is a cost for taking a different action (compared to the previous ...
user avatar
  • 31
1 vote
0 answers
72 views

How to define a Reinforcement Learning episode when dealing with dataset of customer's features?

so one of friends sent me some problem he was working on lately and I couldn't help but to wonder how could it be solved using Q-learning. The statement is as follows : Given the following datasets, ...
user avatar
  • 11
2 votes
0 answers
46 views

Preprocessing data for the learning step

I am currently reading "Human level control through deep reinforcement learning" and I came across the algorithm in the paper. I am confused because the algorithm uses a different notation ...
user avatar
  • 71
-1 votes
1 answer
71 views

Deep Q learning algorithm

Ok I’m a bit confused with this code, what exactly is a time step, isn’t it like when an action is performed,it goes to the next time step, and also, the gradient descent steps is a repeat until ...
user avatar
2 votes
1 answer
189 views

In a DQN, can Prioritized Experience Replay actually perform worse than a regular Experience Replay?

I've written a Double DQN-based stock trading bot using mainly time series stock data. I've recently upgraded my Experience Replay(ER) code with a version of Prioritized Experience Replay (PER) ...
user avatar
  • 31
3 votes
2 answers
318 views

Can an agent be trained in a completely random environment (the rules and actions stay the same)

I am specifically talking about the FrozenLake example from openAI gym to illustrate my question. https://gym.openai.com/envs/FrozenLake-v0/ The way I understand how Q-Learning or DQL works on the ...
user avatar
  • 203
0 votes
1 answer
105 views

Consider more than just a single state-action pair before giving a reward in deep q learning?

TLDR If the environment is rather complex (momentum, velocity, ...), but the action space is rather easy (left, right), is it not better to take multiple decisions into account before giving a reward ...
user avatar
  • 203
2 votes
1 answer
93 views

Q-learning update formula

I am a beginner and it is my first question. I know that Q-learning update equation is: $Q(s_t, a_t) = Q(s_t, a_t)+α(r_{t+1} +γ·max_AQ(s_{t+1}, a_t)−Q(s_t, a_t))$ But in some of the researches it is ...
user avatar
0 votes
1 answer
59 views

Q-Learning [Sutton]: why random variable in formula

Sutton et al. use throughout their book Reinforcement Learning capital letters to describe random variables. At page 131 they introduce Q-Learning. $Q(S_t,A_t)\leftarrow Q(S_t,A_t) + \alpha [R_{t+1} ...
user avatar
  • 3
1 vote
0 answers
250 views

Expected SARSA, SARSA and Q-learning

I would much appreciate if you could point me in the right direction regarding this question about targets for approximate ...
user avatar
  • 111
1 vote
1 answer
466 views

Deep Q Learning best practice

I'm new in deep q-learning and I have understood its main concepts and I'm trying to solve problems with DQL. The problem is that I don't know how to initialize some key values (AKA hyperparameters) ...
user avatar
  • 119
0 votes
1 answer
23 views

Small difference of q-function between different actions for the same state

I am trying out reinforcement learning using Q-learning. The data come from some made-up equations so I have infinite number of data. One thing that troubles me is after I learn the Q-function, I use ...
user avatar
  • 1,643
3 votes
1 answer
603 views

DQN - agent doesn't improve policy

I have a simple grid environment. The player is in the upper left corner and it's goal is to get to lower right corner. The player receives +0.2 points for moving in the direction of the goal, -0.2 ...
user avatar
2 votes
1 answer
904 views

Delayed Rewards in Reinforcement Learning

I have an MDP where the rewards are delayed for six steps as follows: The reward from action at time t is received when the action at time t+6 is taken. The reward from action at time t+1 is ...
user avatar
  • 671
0 votes
0 answers
44 views

How to use data from other policies in order to find optimal policy in model-free rl?

I am struggling to understand whether experience from one policy can be used to find optimal policy. Suppose that I have gathered many data (state, action, reward, next_state) by following random ...
user avatar
  • 671
1 vote
1 answer
1k views

Q-learning agent stucks in an infinite loop

I am simulating a mouse to find a cheese on an empty table. I randomly put a cheese on the table and let the mouse find the cheese without falling off the table. The problem is, in test part, agent ...
user avatar
  • 435
3 votes
1 answer
61 views

Rationale behind Q-learning

I am reading Sutton Barto on Reinforcement Learning. I understand that $TD(\lambda)$ methods propose better performance than Monte Carlo methods, with TD methods combining advantages of Dynamic ...
user avatar
  • 563
3 votes
1 answer
488 views

Reward attribution in deep q learning and texas holdem poker

I’m having issues with reward attribution in poker using deep q learning. Multiple actions will yield one reward, but the reward is only known at the end of the hand, not before. I have built a gym ...
user avatar
  • 301
1 vote
1 answer
48 views

Why not sample action from Q values?

When collecting experience from which to estimate a Q(s,a) function, one common technique in the literature is to follow an epsilon greedy-strategy. In this strategy, the agent selects a random action ...
user avatar