# References Request (Least-Squares Estimates for non i.i.d. Processes)

I am interested in suggestions concerning possible applications/problems within applied statistics with respect to estimates of least-squares for non-stationary designs. In particular, I would like to know if there are current problems in statistics in which it is important to approach the best average response $Y$ from an possibly non-stationary (but, say, independent and bounded) set of data $(X_{1},Y_{1}),\dots,(X_{n},Y_{n})$ in $\mathbb{R}^{d}\times\mathbb{R}$.

In more precise terms, consider the following observation: if $(X_{1},Y_{1}),\dots,(X_{n},Y_{n})$ is a sequence of (possibly non i.i.d.) data in a statistical experiment, then the least squares approximation $$f^{*}:=\arg\min_{f \in \mathcal{F}} \frac{1}{n}\sum_{k=1}^{n}|f(X_{k})-Y_{k}|^{2} \,\,\,\,\,\,\,\,\,\,(1)$$ within a family $\mathcal{F}$ of functions $\mathbb{R}^{d}\to\mathbb{R}$ is the natural "simultaneous" estimator in $L^{2}$ of the conditional expectations of the response variable given the explanatory variable. This is, (1) is the natural empirical estimate of $$\arg\min_{f\in\mathcal{F}}\frac{1}{n}\sum_{k=1}^{n}E(Y_{k}-f(X_{k}))^{2}= \arg\min_{f\in\mathcal{F}}\frac{1}{n}\sum_{k=1}^{n}E(E[Y_{k}|X_{k}]-f(X_{k}))^{2}.\,\,\,\,\,(2)$$

Note that in the i.i.d. case, this leads exactly to the classical least squares regression problem, because $E[Y_{k}|X_{k}]$ does not depend on $k$. Note also that, if $X_{k}$ is stationary (perhaps dependent) and the expectation of $Y_{k}$ given $X_{k}=x$ does not depend on $k$, the setting is still an instance of the classical least-squares problem (for the same reason). Possible ramifications follow easily considering for example the case in which the response variables $E[Y_{k}|X_{k}]$ converge in some sense.

The question therefore is: do you know of any interesting applications/references in which the problem (2) has current relevance? (preferably under the independent, non identically distributed case to begin with, but also in situations of dependence).

Reason for this request: In short: I have been working with some colaborators on the problem of non-stationary least squares regression towards a certain kind of particular Markovian evolution related to Monte Carlo methods.

It turns out that our results so far, if correct, seem to address several intermediate cases along of which we have been wondering whether there are applications of relevance within the community devoted to these methods. This would hopefully give us some useful set of assumptions to further test our results.

Until now I have seen quite a number of articles in which the problem of estimating conditional expectations is addressed via kernel methods for nonindependent, and sometimes nonstationary evolutions. Time Series Analysis, in particular, seems to be an area where these problems are important.

For some reason, nonetheless, the classical least-squares method seems to have not been very explored in that direction. My mathematical intuition tells me that these results can be of great interest also at the practical level and, as pointed out, we would like to confirm this and to even address other existing problems in the statistical community.

• title says non-iid, text says non-stationary ... – kjetil b halvorsen Mar 19 at 3:22