2
$\begingroup$

I would like to design the optimal task distribution system using Reinforcement learning. The best advantage of Reinforcement learning compared to traditional Dynamic programming is that it is not needed to know the information of dynamics such as transition probability. From what I studied for the last days, Reinforcement learning learned that information from experience. But what does it exactly mean learning from experience?

For example, using what I want to design, We don't have exact data sets of task demand from service centers, but we know the distributions of task demand for them. In this case, every iteration, the next states, and the rewards can be random variables that we don't know exactly what could happen. But, because we know the distribution of task demand, we may be able to design following that distribution function in order to decide the next states and the rewards.

Is it an available framework? Is it okay that the next states and the rewards (s',r) are different depend on iteration even though the same state and action (s,a)? In other words, one case can be (s1,a1) -> (s3,r1) but another case can be (s1,a1) -> (s5,r2).

Improve this question
$\endgroup$
2
  • $\begingroup$ I am little bit lost in what is the very essence of your problem. My understanding is that the state of your environment can be described as a vector that expresses the proportion of types of tasks in different distribution centers. For example, if we have 2 distribution centers and 3 types of tasks, you don't have detailed data, but just $s_{i,j}$ where $i=1,2$ and $j=1,2,3$ and $s_{i,j}$ is the number of tasks at distribution center $i$ and task $j$. Is my understanding correct? $\endgroup$
    – Karel Macek
    Commented Sep 29, 2018 at 4:04
  • $\begingroup$ Yes, exactly correct what you described! In that assumption, can you give some ideas to my questions? $\endgroup$
    – Rachel
    Commented Sep 30, 2018 at 23:49

2 Answers 2

Reset to default
2
$\begingroup$

Is it an available framework? Is it okay that the next states and the rewards (s',r) are different depend on iteration even though the same state and action (s,a)?

Yes and yes. From what you described it sounds like it can be implemented using Q learning in a system like Q(S, A) = Q(S, A) + \alpha * [R + \gamma * Q({S}', A) - Q(S,A)], where R is your defined reward.

In detail, you can define your own functions on how transferring from one state to another happens, and also set limits on what actions you can take (e.g. if my states are prices, I can set state transfer function as price dynamics. For actions: for example, if you're optimizing inventory clearance, you can only clear 10 at maximum each time etc).

But what does it exactly mean learning from experience?

In training your reinforcement system, say you're training a Q table.The more times you train the system, the more experience you have until the optimal solution converges (but of course you'll need to have everything needed to calculate reward, e.g. historical data). Hence in the end you get a set of optimal actions under historical states ready to be used in out of sample testing.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks for answers!! from what you said at 2nd question, then do we need to store past states data to decide next state as sampling? $\endgroup$
    – Rachel
    Commented Oct 1, 2018 at 14:11
  • $\begingroup$ @RachelJung if I understand your question correctly, you mean if past state data is needed to determine next step of state right? If your states are sequential, say something like St = alpha*St-1, or robot moved to left or right following by up move etc, then yes we need to store the past states (but most likely only the latest one). $\endgroup$
    – numerairX
    Commented Oct 1, 2018 at 14:34
  • $\begingroup$ Thank you! In other words, for example, state A can transit to state B or state C, which we don't know exactly how often this transition to B or C from A. and We do online learning, which is learning the next state as the data comes in. The agent didn't know the state transition probability, but it will learn from experience that feedbacked from the environment, so the agent ends up learning its underlying transition probability as the value function of states. Does it make sense? $\endgroup$
    – Rachel
    Commented Oct 1, 2018 at 15:26
  • $\begingroup$ @RachelJung it makes sense and it is possible if you define to learn a transfer function, however keep in mind that the end goal of using reinforcement is get maximized long term reward, in this setting, if estimating parameters of transfer function helps the end goal, do it because you have a well defined expectation to maximize. However if you just want to estimate the probability distribution of the state transfer process, imo it should be done before you use this system. Hope that helps. $\endgroup$
    – numerairX
    Commented Oct 1, 2018 at 15:40
1
$\begingroup$ 
Is it an available framework?

Yes, many people use RLlib and openAIgym in python or Ray in java to implement RL algorithms. I know a little about the python ones so I'll mention those in my answer. What you are describing sounds like a standard algorithm from Sutton and Barto which is implemented in the frameworks above. You may need to develop your own openAIgym environment if your problem is a non-standard problem formulation. The environment is a new class which a specified API (collection of functions you must implement. Here is a link to the doc https://gym.openai.com/docs/#environments

But what does it exactly mean learning from experience?

The learned policy (mapping from states to actions) is not known a priori but is learned by (and intelligent) trial and error.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.