Reinforcement Learning in Real Life/Practical Terms

Question

In every day life, it seems that we all have various habits and actions that we perform. For example, we wake up and check our email/facebook etc. on our phones. We don't look at are current state right now, and consider the values of all the possible future trajectories. We basically choose the action that maximizes our "reward" at our current state.

Question. Is it practical to randomly initialize actions $a \in A$, states $s \in S$ and a policy $\pi(a|s)$ and update this according to some algorithm (e.g. REINFORCE, exploration, etc.) to achieve some desired goal in your life? This could be done, for example, by uniformly sampling a random number in the interval $(0,1)$ and acting according to a policy $\pi(a|s)$. For example, suppose your goal is to get married, get a new house etc. What would be appropriate reward functions in this case? Reward is usually defined as the immediate reward plus the discounted cumulative reward. But is this the right definition for practical problems like marriage/dating, buying a house, etc.? Would you define reward as $R_{\text{total}} = R_{\text{marriage}}+R_{\text{buying house}}$ in our example, where each of those individual rewards are typical immediate plus discounted rewards and try to maximize that? Or would it be better to maximize $R_{\text{marriage}}$ and $R_{\text{buying house}}$ individually?

Answer 1

While human behavior is extremely complex and probably can't be described by any simple model, there are good attempts to model and explain common behaviors. You might be interested is the Rescorla-Wagner model of classical conditioning. With that in mind let's examine some of your questions.

We don't look at are current state right now, and consider the values of all the possible future trajectories. We basically choose the action that maximizes our "reward" at our current state.

I don't think this is entirely true. For example if i'm at the bank, the action which greedily maximizes reward is to conduct a bank robbery, but this probably wouldn't end up well for me. So even though I'm not consciously enumerating through every possible action I might take, that doesn't mean there isn't a mechanism in the brain which selects actions considering the cumulative return of the entire future trajectory.

The fact that humans are often nearsighted when it comes to decision making can often be explained by hyperbolic discounting rather than true lack of foresight.

Paraphrasing your other question:

Would it be possible to run your life according to some reinforcement learning algorithm?

I would argue that you are already doing this, and whatever learning algorithm humans use is far more powerful and flexible than anything else we have.

Every single time you make a decision (to eat that tub of ice cream or not -- to buy that new car or a used one), you are implicitly deciding one action or another, hopefully based on some reasonable evaluation of which one will result in a higher cumulative return. Perhaps for most decisions you don't explicitly compute out how many units of cumulative return each action will likely result in -- although if you've ever played poker, this is crucial! -- but the computation is happening nonetheless.

As for the question of what the reward function should be -- well, that is the domain of philosophy.