I'm new in machine learning topics and I've problem in modeling my environment which has multi parameters with different value ranges and a few actions to perform when value of each parameter is not in normal range.

Each parameter of my environment has to threshold (minTh and maxTh), therefor ranges of its value fit in three region, one is between threshold that is normal region and two is beyond thresholds that are violated regions.

We measure each parameter's value, and when it goes to violated value must take an action to normalize parameter's value. But, the action may effect the others value too (bad or good effect).

My aim is best action selection for each of this situations and make better decision next times a violation occurs with learning effect of actions on parameters.

I think I may choose Markov Decision Process (or its successors) technique to model my environment. My problem is how to model States and Reward function for this environment.

For Example:

Consider we have two rooms and our parameters is temperature and moisture of these rooms(4 parameters) each have its own thresholds and acceptable values. (for example room One acceptable temperature is between 18 and 30)

I have some actions like turning on or off of air conditioner or heater or etc. in each room, that must perform when value of a parameter is goes to violated region.(example room one temp be 10)

How can I model this environment if I choose Markov Decision Process? or is there any better technique for modeling and solving this problem?

  • 1
    $\begingroup$ Please spell out your acronyms. What is "MDP"? $\endgroup$ Feb 3, 2016 at 14:00
  • $\begingroup$ I mean Markov Decision Process $\endgroup$ Feb 3, 2016 at 14:02

1 Answer 1


Yes, the problem you're describing can be modeled as a Markov decision process, particularly one with a continuous domain. (Meaning, a continuous state space.) Your room example would have a four-dimensional state space, corresponding to each room's temperature and humidity.

A standard example of this type of problem is the mountain car, described in Sutton and Barto's reinforcement learning text, which might help you understand the problem in fewer dimensions. This tutorial may also help, particularly if you can follow Java.

To your question about the reward function, you can answer this by asking what costs are associated with dipping below this range. This doesn't need to be exact to lead to the correct solution. Optimal policies are invariant to scaling and shifting operations performed on the reward function. Meaning, if the cost of going outside the parameters is uniform, a reward of $-1$ in the out-of-bounds states should suffice.

(In other applications, the reward function is just an arbitrary means of encoding a goal. E.g., the goal state has a reward of, say, $100$, with other rewards uniformly $0$ or a small negative number.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.