0
$\begingroup$

Imagine a rigid solid, just subjected to an external upwards effort and gravity. I am trying to calculate by RL the effort needed along time so that the solid reaches and maintains at a certain height.

I am using: - heights of each moment as states - three possible actions (positive fixed effort, negative fixed effort and 0 effort).

My doubt consists in how to create the transition matrix. I am following a value iteration policy, and it is defined as: the probability of getting from a state s to s' with an action a. If I'm using three different actions, do I need to create three different matrix? I'm a bit lost.

$\endgroup$
0
$\begingroup$

You might be mixing up some terms. If you define a transition matrix, it's not RL anymore. Value iteration also only works in discrete spaces & the transition matrix corresponds to states. You can either discretize your states (which might make sense depending on your model) or look into alternative methods that can solve domains with continuous states and discrete actions (possibly optimal control methods such as DDP but they usually are completely continuous). You can also solve it with any RL method via simulation but then you don't specify any transition model.

$\endgroup$
  • $\begingroup$ Thanks for the reply. However, maybe I explained myself wrong. What I want to achieve is present in the second page, first line of this document. Isn't that RL? Specifically, I am trying to implement this paper, which also uses this strategy (Section 4.1) $\endgroup$ – galtor Apr 4 '17 at 6:04
  • $\begingroup$ It is only RL if the transition model is unknown. The second paper you link is model-based RL: The transition model is unknown but learned based on samples. The exact process is barely described (and usually, you wouldn't learn this as a matrix, but whatever works for the application, I guess) That also tells you where your transition matrix comes from. How you represent the matrix is up to you but you can make it work by having one matrix with |S||A| rows and |S| columns. $\endgroup$ – DaVinci Apr 7 '17 at 22:15
  • $\begingroup$ I'm starting to understand. I think that my situation is described in this question, second answer. I don't know the probs., but I know the dynamic model. And so, I think that Q matrix is useful. I don't understand the measures of the matrix you state. How many rows do you refer? Is like this, in page 6? $\endgroup$ – galtor Apr 11 '17 at 8:21
  • $\begingroup$ If you have a model based approach, where you learn the dynamics model, you need to represent it. This could be as a matrix which has all the probabilities or as a function approximator. The document you link does the former, usually you'd do the latter. If you represent it as a matrix, you choose the row based on state and action and the column based on the next state. You don't need 3 matrices like you state in the question but it's really an implementation detail. $\endgroup$ – DaVinci Apr 14 '17 at 20:39
  • $\begingroup$ I think I got clearer. I'm going to approve the first answer and open a new one in case I get stuck. $\endgroup$ – galtor Apr 18 '17 at 20:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.