Translating machine learning problem into regression framework

Question

Suppose I have a panel of explanatory variables $X_{it}$, for $i = 1 ... N$, $t = 1 ... T$, as well as a vector of binary outcome dependent variables $Y_{iT}$. So $Y$ is only observed at the final time $T$ and not at any earlier time. The fully general case is to have multiple $X_{ijt}$ for $j=1...K$ for each unit $i$ at each time $t$, but let's focus on the case $K=1$ for brevity.

Applications of such "unbalanced" $(X, Y)$ pairs with temporal correlated explanatory variables are e.g. (daily stock prices, quarterly dividends), (daily weather reports, yearly hurricanes) or (chess position features after each move, win/loss outcome at the end of the game).

I am interested in the (possibly non-linear) regression coefficients $\beta_t$ for doing prediction of $Y_{it}$, knowing that in the training data, given early observations of $X_{it}$ for $t < T$, it leads to final outcome $Y_{iT}$

$\hat{Y}_{it} = f(\sum_{k=1}^{t} X_{ik} \beta_k), \quad t = 1 ... T$

Coming from an econometrics background, I haven't seen much regression modelling applied to such data. OTOH, I have seen the following machine learning techniques being applied to such data:

1. doing supervised learning on the entire data set, e.g. minimizing

$\sum_{i,t}\frac{1}{2}(Y_{it} - f(X_{it} \beta_t))^2$

by simply extrapolating/imputing the observed $Y$ to all previous points in time

$Y_{it} \equiv Y_{iT}, \quad t = 1... T-1$

This feels "wrong" because it will not take into account the temporal correlation between the different points in time.

2. doing reinforcement learning such as temporal-difference with learning parameter $\alpha$ and discount parameter $\lambda$, and recursively solving for $\beta_t$ through back-propagation starting from $t=T$

$\Delta \beta_{t} = \alpha (\hat{Y}_{t+1} - \hat{Y}_{t}) \sum_{k=1}^{t} \lambda^{t-k} \nabla_{\beta} \hat{Y}_{k}$

with $\nabla_{\beta} \hat{Y}$ the gradient of $f()$ with respect to $\beta$.

This seems more "correct" because it takes the temporal structure into account, but the parameters $\alpha$ and $\lambda$ are kind of "ad hoc".

Question: is there literature on how to map the above supervised / reinforcement learning techniques into a regression framework as used in classical statistics / econometrics? In particular, I'd like to be able to estimate parameters $\beta_{t}$ in "one go" (i.e. for all $t=1...T$ simultaneously) by doing (non-linear) least-squares or maximum-likelihood on models such as

$Y_{iT} = f(\sum_{t=1}^T X_{it} \beta_{t}) + \epsilon_{i}$

I'd also be interested to learn whether the temporal difference learning meta-parameters $\alpha$ and $\lambda$ could be recovered from a maximum-likelihood formulation.

Answer 1

The description of the problem is not entirely clear to me so I try to guess some assumptions. If this does not answer your question, it might at least help to clarify the issues further.

The first thing which is not clear to me is the data you want to base your prediction on. If you would like to predict $Y_T$ based on observed data until $t<T$ then a recursive approach as in your method 2. does not make sense since this would use future data, i.e. $X_\tau$ with $\tau>t$.

Second you do not state what the properties of your predicted $Y_t$ shall be. Generally, given information $X_1,\ldots, X_t$ at time $t<T$ the conditional expectation $Y_t=\text{E}[Y_T \mid X_1,\ldots, X_t]$ is the "best predictor" of $Y_T$ in the L2 sense. In case you really want to predict the conditional expectation ordinary least squares is the method of choice for practical estimation.

Furthermore, I do not understand your remark about the correlations not being reflected by the regression based on the $X_1, \ldots, X_t$. This incorporates everything you know until $t$ including the correlations between your observations.

So summing up and phrasing this as an answer: If you want to make an optimal prediction in the L2 sense, based only on data observed until $t<T$ you can use least squares regression.

Answer 2

The advantage of temporal differences is that they allow you to learn from incomplete episodes. So, sequences where you haven't got to the the final Y can be still be used to fit the model; subsequent estimates are used instead. The effect is similar to hidden data imputation; implicitly you are imputing the remainder of the sequence according to your current model.
Temporal difference models are normally trained by stochastic gradient descent. $\alpha$ controls the learning rate. Too high and the method will diverge. Too low and convergence to a local optimum will be very slow. But convergence should always be to the same model.
Here, $\gamma$ controls the relative effort given to predictions depending on how far they are from the end of a sequence. Because these sequences are finite in length, you can set this to $\gamma=1$ , to put the same weight on all estimates.