Suppose I have a panel of explanatory variables $X_{it}$, for $i = 1 ... N$, $t = 1 ... T$, as well as a vector of binary outcome dependent variables $Y_{iT}$. So $Y$ is only observed at the final time $T$ and not at any earlier time. The fully general case is to have multiple $X_{ijt}$ for $j=1...K$ for each unit $i$ at each time $t$, but let's focus on the case $K=1$ for brevity.

Applications of such "unbalanced" $(X, Y)$ pairs with temporal correlated explanatory variables are e.g. (daily stock prices, quarterly dividends), (daily weather reports, yearly hurricanes) or (chess position features after each move, win/loss outcome at the end of the game).

I am interested in the (possibly non-linear) regression coefficients $\beta_t$ for doing prediction of $Y_{it}$, knowing that in the training data, given early observations of $X_{it}$ for $t < T$, it leads to final outcome $Y_{iT}$

$\hat{Y}_{it} = f(\sum_{k=1}^{t} X_{ik} \beta_k), \quad t = 1 ... T$

Coming from an econometrics background, I haven't seen much regression modelling applied to such data. OTOH, I have seen the following machine learning techniques being applied to such data:

  1. doing supervised learning on the entire data set, e.g. minimizing

$\sum_{i,t}\frac{1}{2}(Y_{it} - f(X_{it} \beta_t))^2$

by simply extrapolating/imputing the observed $Y$ to all previous points in time

$Y_{it} \equiv Y_{iT}, \quad t = 1... T-1$

This feels "wrong" because it will not take into account the temporal correlation between the different points in time.

  1. doing reinforcement learning such as temporal-difference with learning parameter $\alpha$ and discount parameter $\lambda$, and recursively solving for $\beta_t$ through back-propagation starting from $t=T$

$\Delta \beta_{t} = \alpha (\hat{Y}_{t+1} - \hat{Y}_{t}) \sum_{k=1}^{t} \lambda^{t-k} \nabla_{\beta} \hat{Y}_{k}$

with $\nabla_{\beta} \hat{Y}$ the gradient of $f()$ with respect to $\beta$.

This seems more "correct" because it takes the temporal structure into account, but the parameters $\alpha$ and $\lambda$ are kind of "ad hoc".

Question: is there literature on how to map the above supervised / reinforcement learning techniques into a regression framework as used in classical statistics / econometrics? In particular, I'd like to be able to estimate parameters $\beta_{t}$ in "one go" (i.e. for all $t=1...T$ simultaneously) by doing (non-linear) least-squares or maximum-likelihood on models such as

$Y_{iT} = f(\sum_{t=1}^T X_{it} \beta_{t}) + \epsilon_{i}$

I'd also be interested to learn whether the temporal difference learning meta-parameters $\alpha$ and $\lambda$ could be recovered from a maximum-likelihood formulation.

  • $\begingroup$ Could you clarify the formulation in the third paragraph? You write that you want to predict $Y_{iT}$ from $X_{it}$, $t < T$, but the following formula suggests that you want to predict $Y_{it}$. $\endgroup$
    – NRH
    Sep 4, 2015 at 19:39
  • $\begingroup$ @NRH actually, I only observe $Y_{iT}$, but what I've seen in the literature on supervised learning is that they impute the unobserved $Y_{it}$ to be equal to $Y_{iT}$ and then do the fitting to actually explain this fake $Y_{it}$ from $X_{it}$ (this is done in game playing applications, where an evaluation function for each position is fitted on the game's final result). Sorry if this was not clear from my initial formulation. In any case, $\hat{Y}_{it}$ would be the predicted "outcome" (in game applications) given observed events $X_{it}$. $\endgroup$ Sep 4, 2015 at 19:50
  • $\begingroup$ I understand the setup and what you observe, but your formulation in the question is unclear. Do you want to train a model for predicting $Y_{iT}$ as you write in words, or do you want to train a model for predicting $Y_{it}$ for all $t$ as the formulas suggest? Maybe it's just a typo. When you write "… prediction of $Y_{iT}$ …" do you mean "… prediction of $Y_{it}$ ..."? $\endgroup$
    – NRH
    Sep 6, 2015 at 12:21
  • $\begingroup$ its not clear why you want to do this. If you can explain the actual practical application you might get clearer answers. In general, the best prediction for each timespan will just be doing a regression of $Y_T$ on the available data $X_1,\dots,X_t$ separately for each t. It is not obvious that a simultaneous approach has any benefit. I think you have to specify the statistical model for your data set and then maybe the benefits are clearer. $\endgroup$
    – seanv507
    Sep 6, 2015 at 15:16
  • $\begingroup$ @NRH, yes, I want to predict $Y_{it}$ from $X_{it}$ knowing that it leads to outcome $Y_{iT}$ in the training data, in order to take optimal actions for test data where I also observe $X_{it}$ but have not yet observed the outcome. Will update my formulation. $\endgroup$ Sep 6, 2015 at 20:06

2 Answers 2


The description of the problem is not entirely clear to me so I try to guess some assumptions. If this does not answer your question, it might at least help to clarify the issues further.

The first thing which is not clear to me is the data you want to base your prediction on. If you would like to predict $Y_T$ based on observed data until $t<T$ then a recursive approach as in your method 2. does not make sense since this would use future data, i.e. $X_\tau$ with $\tau>t$.

Second you do not state what the properties of your predicted $Y_t$ shall be. Generally, given information $X_1,\ldots, X_t$ at time $t<T$ the conditional expectation $Y_t=\text{E}[Y_T \mid X_1,\ldots, X_t]$ is the "best predictor" of $Y_T$ in the L2 sense. In case you really want to predict the conditional expectation ordinary least squares is the method of choice for practical estimation.

Furthermore, I do not understand your remark about the correlations not being reflected by the regression based on the $X_1, \ldots, X_t$. This incorporates everything you know until $t$ including the correlations between your observations.

So summing up and phrasing this as an answer: If you want to make an optimal prediction in the L2 sense, based only on data observed until $t<T$ you can use least squares regression.

  • $\begingroup$ in the training data, I want to use the fact that a given $X_{it}$ observation will statistically lead to outcome $Y_{iT}$ in order to predict $\hat{Y}_{it}$ for test data where I don't observe $Y_{iT}$ until afterwards. If e.g. you know that after 3 windy days it will likely rain on day 7, you want to use that information to tell people to bring umbrellas after the weekend after a few windy days before. $\endgroup$ Sep 6, 2015 at 20:23

The advantage of temporal differences is that they allow you to learn from incomplete episodes. So, sequences where you haven't got to the the final Y can be still be used to fit the model; subsequent estimates are used instead. The effect is similar to hidden data imputation; implicitly you are imputing the remainder of the sequence according to your current model.
Temporal difference models are normally trained by stochastic gradient descent. $\alpha$ controls the learning rate. Too high and the method will diverge. Too low and convergence to a local optimum will be very slow. But convergence should always be to the same model.
Here, $\gamma$ controls the relative effort given to predictions depending on how far they are from the end of a sequence. Because these sequences are finite in length, you can set this to $\gamma=1$ , to put the same weight on all estimates.

  • $\begingroup$ This does not really answer the question: e.g. how can the $\alpha$ and $\gamma$ parameters be set optimally in a maximum-likelihood framework? $\endgroup$ Sep 4, 2015 at 12:46
  • $\begingroup$ $\alpha$ controls the speed of convergence but should has no effect on the final model or the likelihood of that model. In practice, I set it by trial and error. You have to set $\gamma$ as it controls the relative importance of short term versus long term predictions if the same parameters are used across short and long predictions. That will be application specific depending on what you want to do with the predictions. $\endgroup$
    – nsweeney
    Sep 4, 2015 at 14:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.