Skip to main content
15 events
when toggle format what by license comment
Dec 15, 2018 at 6:00 history tweeted twitter.com/StackStats/status/1073820081002561537
Dec 13, 2018 at 6:59 comment added DaVinci In the general case, the value function with a discount factor of 1 is ill-defined. There may be infinitely long sequences that accrue infinite cumulative reward, hence there can be no guarantees for a discount factor of 1. It may still work, however. The learning rate, e.g. alpha in these update-rules: en.wikipedia.org/wiki/Q-learning, is essential in the case where the transitions aren't deterministic. You want to average the future rewards you are getting. but with a learning rate of 1 you are always using the latest observation which doesn't converge.
Dec 12, 2018 at 21:31 comment added gung - Reinstate Monica This is being automatically flagged as low quality, probably because it is so short. At present it is more of a comment than an answer by our standards. Can you expand on it? You can also turn it into a comment.
Dec 12, 2018 at 21:25 vote accept Most Wanted
Dec 12, 2018 at 21:18 comment added Sycorax Could you elaborate about why using a discount factor of 1 and a learning rate will cause this problem?
Dec 12, 2018 at 21:07 answer added Most Wanted timeline score: 4
Dec 12, 2018 at 15:44 comment added ljeabmreosn Since all of the rewards are non-negative, you can convert all of the rewards r for each state s (excluding the "end" state) to r'(s) = -d/(r(s)+c) for positive constants c and d. For example, if c = d = 1 then r'(s) = -1/(r(s)+1).
Dec 12, 2018 at 13:40 comment added Most Wanted Makes a perfect sense. What would you suggest to do? I've tried to set reward at final state to a very high value, but obviously agent will not find best path to it and only will get to that point.
Dec 11, 2018 at 22:06 comment added ljeabmreosn I don't think anything is missing from the code provided. The rewards that are shown don't make much sense. If some of the intermediate states have a higher reward than the desired "end" state, why wouldn't the agent with a maximal policy attempt to stay in the highest reward states? This is similar to trying to find a minimum cost path in a graph with a negative cost cycle.
Dec 11, 2018 at 19:59 comment added Most Wanted I need a Bellman equation to maximize cumulative result for my rewards? Where should I apply it? Is it missing from the code above or it should be done on a policy extraction step?
Dec 11, 2018 at 19:46 comment added Most Wanted ok, discount factor is equal to 1 in my example, because rewards stay the same for each step. Where should I apply my learning rate? Is it different from 1 factor and will influence resulting convergence?
Dec 10, 2018 at 21:45 comment added DaVinci You're missing a learning rate and a discount factor in your Q-learning update rule. Both are necessary for convergence.
Dec 6, 2018 at 16:59 comment added ljeabmreosn What is the "correct result"? According to the rewards that you have assigned, being in the states (2, 0), (2, 1), and (1, 1), is better than getting to the end state. So the policy will continue to stay in those states that have high reward. In order to get the desired result, try making each state (excluding the end state) have a non-positive reward. This way, the policy extracted from value iteration will not get stuck in an infinite loop.
Dec 6, 2018 at 8:15 review First posts
Dec 6, 2018 at 8:23
Dec 6, 2018 at 8:12 history asked Most Wanted CC BY-SA 4.0