Timeline for Value iteration does not converge when using Q learning
Current License: CC BY-SA 4.0
15 events
when toggle format | what | by | license | comment | |
---|---|---|---|---|---|
Dec 15, 2018 at 6:00 | history | tweeted | twitter.com/StackStats/status/1073820081002561537 | ||
Dec 13, 2018 at 6:59 | comment | added | DaVinci | In the general case, the value function with a discount factor of 1 is ill-defined. There may be infinitely long sequences that accrue infinite cumulative reward, hence there can be no guarantees for a discount factor of 1. It may still work, however. The learning rate, e.g. alpha in these update-rules: en.wikipedia.org/wiki/Q-learning, is essential in the case where the transitions aren't deterministic. You want to average the future rewards you are getting. but with a learning rate of 1 you are always using the latest observation which doesn't converge. | |
Dec 12, 2018 at 21:31 | comment | added | gung - Reinstate Monica | This is being automatically flagged as low quality, probably because it is so short. At present it is more of a comment than an answer by our standards. Can you expand on it? You can also turn it into a comment. | |
Dec 12, 2018 at 21:25 | vote | accept | Most Wanted | ||
Dec 12, 2018 at 21:18 | comment | added | Sycorax♦ | Could you elaborate about why using a discount factor of 1 and a learning rate will cause this problem? | |
Dec 12, 2018 at 21:07 | answer | added | Most Wanted | timeline score: 4 | |
Dec 12, 2018 at 15:44 | comment | added | ljeabmreosn |
Since all of the rewards are non-negative, you can convert all of the rewards r for each state s (excluding the "end" state) to r'(s) = -d/(r(s)+c) for positive constants c and d . For example, if c = d = 1 then r'(s) = -1/(r(s)+1) .
|
|
Dec 12, 2018 at 13:40 | comment | added | Most Wanted | Makes a perfect sense. What would you suggest to do? I've tried to set reward at final state to a very high value, but obviously agent will not find best path to it and only will get to that point. | |
Dec 11, 2018 at 22:06 | comment | added | ljeabmreosn | I don't think anything is missing from the code provided. The rewards that are shown don't make much sense. If some of the intermediate states have a higher reward than the desired "end" state, why wouldn't the agent with a maximal policy attempt to stay in the highest reward states? This is similar to trying to find a minimum cost path in a graph with a negative cost cycle. | |
Dec 11, 2018 at 19:59 | comment | added | Most Wanted | I need a Bellman equation to maximize cumulative result for my rewards? Where should I apply it? Is it missing from the code above or it should be done on a policy extraction step? | |
Dec 11, 2018 at 19:46 | comment | added | Most Wanted | ok, discount factor is equal to 1 in my example, because rewards stay the same for each step. Where should I apply my learning rate? Is it different from 1 factor and will influence resulting convergence? | |
Dec 10, 2018 at 21:45 | comment | added | DaVinci | You're missing a learning rate and a discount factor in your Q-learning update rule. Both are necessary for convergence. | |
Dec 6, 2018 at 16:59 | comment | added | ljeabmreosn |
What is the "correct result"? According to the rewards that you have assigned, being in the states (2, 0) , (2, 1) , and (1, 1) , is better than getting to the end state. So the policy will continue to stay in those states that have high reward. In order to get the desired result, try making each state (excluding the end state) have a non-positive reward. This way, the policy extracted from value iteration will not get stuck in an infinite loop.
|
|
Dec 6, 2018 at 8:15 | review | First posts | |||
Dec 6, 2018 at 8:23 | |||||
Dec 6, 2018 at 8:12 | history | asked | Most Wanted | CC BY-SA 4.0 |