5
$\begingroup$

In reinforcement learning, there is a state $s_t$, an action $a_t$, and a policy $\pi(a|s)$ that maps states to the Probability Distribution Function (PDF) of actions. The goal is to choose the optimal action given the state; we can denote the optimal action $a_t^*$, and the corresponding optimal policy that selects this action is $\pi^*(a|s)$. Note that $a_t^*$ is typically unobservable, even after time $t$.

In time-series we have a dependent variable $y_t$, explanatory variables $x_t$, and a model (typically parameterized by $\theta$) linking them, typically including a residual term, denoted $y_t = g(x_t, u_t | \theta)$. If the distribution of $u_t$ is known, and $\theta$ is observable, then the model yields the conditional PDF $f(y_t|x_t;\theta)$. This is particularly straightforward when $g$ is linear. Since $u_t$ is latent, our model provides predictions of $y_t$, which we denote $\hat{y}_t$. In practice, we don't know $\theta$, and may not know $f$, so at any given time we can only observe $\hat{f}(\hat{y}_t|x_t;\hat{\theta})$, which is an estimate of the true Conditional PDF of $y_t$.

Let $x_t = s_t$, $y_t = a_t^*$, $\hat{y}_t = a_t$, $f(y_t|x_t;\theta) = \pi^*(a|s)$, and $\hat{f}(\hat{y}_t|x_t;\hat{\theta}) = \pi(a|s)$. Given these equivalencies, it appears that the time-series and reinforcement learning frameworks are equivalent, with the key difference being that in time-series, typically $y_t$ is observable, but in this situation it would need to be treated as latent.

My Question: Given latent $y_t$, are the above two frameworks equivalent? If so, why does the standard introductions to reinforcement learning (e.g. Sutton and Barto's textbook), not discuss topics like the Kalman Filter and EM algorithm? (given these topics are suited to time-series with latent variables)

Background: My background is in time-series, and I'm currently reading Sutton and Barto's textbook and trying to get my head around Reinforcement Learning. Perhaps trying to cram it into a time-series framework is a bad idea, but given the whole thing seems to be based on Markov transition matrices, I can't escape the feeling that it fits there rather neatly. So I thought I'd ask here and see if anyone else has thought about this.

$\endgroup$

1 Answer 1

5
+50
$\begingroup$

While there are some conceptual parallels, there are also fundamental differences that might explain why time-series methodologies like the Kalman Filter and EM algorithm are not typically discussed in standard RL introductions. Here are some key points of differentiation:

In time-series analysis, the goal is often to predict future observations of a variable based on past observations and explanatory variables. In RL, the goal is to learn a policy that maximises cumulative reward over time. The focus in RL is on learning optimal actions, which is different from predicting a variable.

In RL, the concept of optimality is tied to the notion of rewards and value functions, which represent the expected cumulative reward that can be obtained from a state or state-action pair. This is inherently different from time-series analysis, where optimality might be tied to best explaining the variance in the data or best predicting future observations.

RL often deals with environments that have complex, unknown dynamics that can only be understood through interaction. In many time-series applications, the system dynamics are assumed to be known and modelled directly.

In time-series, latent variables often represent unobserved factors that affect the observed data, and methods like the Kalman Filter or EM algorithm are used to estimate these factors. In RL, the optimal action a* is unobservable because it is a theoretical construct: the best possible action given complete knowledge of the environment and the future.

RL inherently involves exploration (trying out different actions to discover their effects) and exploitation (choosing the best-known action). Time-series analysis typically does not include a mechanism for exploration, as it usually deals with observed data rather than an environment where one can actively take actions to gather data.

Regarding why the Kalman Filter and EM algorithm are not typically discussed in RL contexts:

The Kalman Filter is specifically designed for linear systems with Gaussian noise. While it has been extended to non-linear systems (e.g., through the Extended Kalman Filter or Unscented Kalman Filter), these methods assume a known model of the system. In RL, the model is often unknown and must be learned, which is a different problem setup.

The EM algorithm is used for maximum likelihood estimation when data are missing or latent. In RL, the problem is not about parameter estimation with incomplete data in the statistical sense, but about learning to make decisions that maximise reward.

While RL and time-series analysis might use some similar mathematical tools (like Markov chains), they are applied in different contexts with different objectives. It's not that one framework is better than the other; they are just suited to different types of problems. That said, there are areas within RL, such as Partially Observable Markov Decision Processes (POMDPs), where concepts from time-series analysis and state estimation are more directly applicable. However, these areas are more specialised and may not be covered in introductory texts.


Edit the address the query in the first comment

You're correct that minimising loss in time-series analysis can be conceptually similar to maximising reward in reinforcement learning (RL) when you interpret the loss function as the negative of the reward function. In time-series analysis, a common goal is to minimise the expected loss (or error) of predictions, such as the difference between predicted values and actual values. This is analogous to maximising expected cumulative reward in RL, where the "loss" could be seen as the negative of the reward.

However, the key distinction lies in the structure and intention of the models:

Objective Functions: In time-series, the loss functions are typically symmetric and penalise overestimation and underestimation equally. In RL, the reward function might not be symmetric and can be structured to encourage certain behaviours over others.

Decision-Making: RL explicitly models decision-making under uncertainty, where actions influence future states and thus future rewards. Time-series models typically do not influence the system they are predicting; they are passive observers.

Cumulative Reward: RL focuses on cumulative reward, which involves a sequence of decisions where each decision impacts future rewards. In contrast, time-series often looks at single-step predictions without considering the long-term impact of those predictions.

Exploration vs. Prediction: RL involves exploration to discover the reward function, while time-series analysis assumes that the underlying process generating the data is stationary and that the future will behave like the past.

While both time-series and RL can be seen through the optimisation lens, they apply to different problem spaces: time-series to forecasting based on observed data, and RL to sequential decision-making where each action affects future observations and rewards. This distinction often necessitates different tools and approaches, which is why you don't commonly see time-series methods like the Kalman Filter or EM algorithm in standard RL introductions.

$\endgroup$
3
  • $\begingroup$ Great answer - thank you +1. I'll hold off on awarding the bounty for another few days just to give others a chance to respond. One point I would query regarding optimality: In time-series, a common goal is to minimize expected loss of predictions. For a sufficiently general interpretation of loss functions, how is this different from maximizing expected cumulative future reward? For example, $L(a_t,a_t^*)=0$ implies maximizing reward, so we're already satisfying one of the basic properties of a loss function. $\endgroup$ Commented Nov 10, 2023 at 1:55
  • $\begingroup$ You're welcome, and no worries. I will update my answer as a response to this comment $\endgroup$ Commented Nov 10, 2023 at 8:44
  • $\begingroup$ Thanks for the update! I've given the tick and the bounty. $\endgroup$ Commented Nov 12, 2023 at 3:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.