I'm having trouble identifying what statistical model or methodology is suited for my application.

My situation is as follows:

I want to create a stock trading agent that trades a single stock-cash pair based on quote data it receives over time.

For simplicity:

  1. If the agent is holding only cash, their available actions are to either buy 1 unit of the stock or do nothing.
  2. If the agent is holding either cash and stock or only stock, their available actions are to either do nothing or sell the unit of stock.
  3. If the agent is holding no cash and no stock, the agent's only available action is to do nothing.

The notation I will use is this:

  • $t_0, t_1 \in \mathbb{R}_{\geq 0}$: time of first and second order book updates, respectively.

  • $\mathbb{B}_0, \mathbb{B}_1 \in ({\mathbb{R}_{\geq 0}})^4$: quote data received at $t_0$ and $t_1$, respectively.

  • $v_0, v_1 \in \mathbb{R}_{\geq 0}$: amount of cash held at $t_0$ and $t_1$, respectively.

  • $w_0, w_1 \in \mathbb{R}_{\geq 0}$: amount of stock held at $t_0$ and $t_1$, respectively.

  • $L(.,.,.)$denotes the liquidation value of the holdings given the current quote data, current held cash, and current held stock

The single period setup for this is as follows:

  1. The agent observes $o_0 = (t_0,\mathbb{B}_0, v_0, w_0)$
  2. The agent chooses an action $a_0^* \in \mathcal{A}_0$ (this decision would likely be informed by a prediction of the time of the next quote, $\hat{t_0}$, a prediction of the next quote, $\hat{\mathbb{B}_{1}}$, and predictions of the cash and stock held at $t_1$ as a result of any $a_0 \in \mathcal{A}_0$, $\hat{v_1}_{|a_0}$ and $\hat{w_1}_{|a_0}$, for all $a_0 \in \mathcal{A}_0$.
  3. The agent observes $o_1 = (t_1,\mathbb{B}_1, v_1, w_1)$
  4. The agent receives $l_0 = L(\mathbb{B}_1, v_1, w_1) - L(\mathbb{B}_0, v_0, w_0)$ (the profit resulting from taking $a_0$ when $o_0$ was observed and then $o_1$ being observed).

My intuition says some sort of Markov Decision Process would be suitable for this, but I'm unsure since even with the largely simplifying assumption (that could be mostly true for small orders) that $a_0$ doesn't affect $\mathbb{B}_1$, we still have to consider that $a_0$ will affect $v_1$ and $w_1$ (e.g. if we decide to buy a unit of stock at $t_0$, that will very likely impact $v_1$ and $w_1$) even if we do something as simple as assuming it's based only deterministically on the current quote, $\mathbb{B}_0$ (that "nothing happens" between $t_0$ and $t_1$ with respect to the quote data), we still have to model this for each potential action.

The goal of this whole process being to find a function of $o_1$ that outputs an action in $\mathcal{A}_0$ that minimizes $l_0$.

Is there a name for a general version of this type of problem that I can look into or a way to simplify parts of it so I can fit it into a more studied framework?

  • 1
    $\begingroup$ It's perfectly OK to have the decision affect the transition probabilities; the model is more complex, but it's still an MDP. $\endgroup$
    – jbowman
    Jan 1 at 22:50
  • $\begingroup$ Great, thank you for the assurance @jbowman ! Do you think that I am perhaps overcomplicating this for an exploratory development environment? Because given a time series of actual quote data, I would still have to simulate what would happen to my holdings over time for any action I choose and during deployment, this would be determined out of my hands by the market dynamics of whatever exchange I choose to trade on (with regards to how the holdings change after an action is taken due to concepts such as slippage)? $\endgroup$
    – QMath
    Jan 2 at 2:05
  • 1
    $\begingroup$ I think it's OK, as a modeling exercise, but I agree with the answer below - don't do something like this for real trades with your money. $\endgroup$
    – jbowman
    Jan 2 at 5:14
  • $\begingroup$ okay, @jbowman , that makes sense, thank you for your honesty, would you say that there is a particular reason that you would advise against trying to implement a system like this for applicational/practical use? or more a combination reasons such as lack of scale/infrastructure, model "incorrectness", etc, $\endgroup$
    – QMath
    Jan 2 at 5:38

1 Answer 1


In general, I won't suggest doing something like this for real trade with your money. This field is enormously competitive with companies with huge data and milli-second real-time advantages.

For the sake of exposition, I don't think you can apply vanilla MDP to this. You can assume Markov property has some hidden factors, but the entire system's stationarity is ridiculous to assume. Look at Reinforcement Learning methods, and you will find some good techniques to start.

Do not do this with your money!

  • $\begingroup$ +1, especially for the warnings! $\endgroup$
    – jbowman
    Jan 2 at 5:15
  • $\begingroup$ Thank you for the insight! Just so I can be clear in my reasoning, is there anywhere that I specifically assumed stationarity of a system? $\endgroup$
    – QMath
    Jan 2 at 5:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.