# Linear regression - iterative approach

I have a single output variable $y$ and a number of inputs $x_1$, $x_2$, etc. These are time series. Each $x_i$ explains the changes in $y$ in specific circumstances, and the goal is to have a linear model that looks like $y=b_1x_1+b_2x_2+...$ The point is that each $x_i$ must be sampled on different criteria specific to its qualities, and its relationship to $y$ to be established only at those situations when that particular $x_i$ is the lead influence on $y$.

My approach is to take one sampling method $s_1$, sample $y$ and $x1$ on it, specific to $x_1$. Do the regression $y$ on $x_1$. Then, take the error $y-b_1x_1$, sample the error on another sampler $s2$, and regress this on $x_2$.

What's the name of this method, is there a regression of this sort? Does it sound like the right solution to my problem?

Due to practical constraints, I want my model to include all $x$s at the same time, rather than switch between different models on the fly.

• By time series, do you mean $y = y(t)$ and $x_i = x_i(t)$? Also, it's not clear what you mean by the statement "each $x_i$ must be sampled on different criteria..." From what I understand based on your description, your approach seems like boosting (en.wikipedia.org/wiki/Gradient_boosting). Commented Dec 9, 2015 at 5:16
• Yes, that's what I mean by time series. Sampled on different criteria means on different clock. Commented Dec 9, 2015 at 5:30
• Can you explain what the training data looks like? It seems like you have a set of $n$ training samples of the form $(y^{(1)}, x^{(1)}_1, x^{(1)}_2, dots), (y^{(2)}, x^{(2)}_1, x^{(2)}_2, \dots), \dots, (y^{(n)}, x^{(n)}_1, x^{(n)}_2, \dots)$ where $y^{(j)}$ and $x^{(j)}_i$s are times series and $x^{(j)}_i(t)$ may or may not be available for some $t$. Is this correct?
– Sobi
Commented Dec 13, 2015 at 5:32

I'm posting as an answer since it's too long for a comment.

If I understand correctly, you have data coming from, let's say, $N$ multiple sensors $x_i^{(t)} \in \mathbb{R}^{T \times N}$ where each sensor is a time series with $T$ data points. You also have a target output variable $y^{(t)} \in \mathbb{R}^T$.

You would like to perform a prediction using a linear model and you would like to adjust the subsequence sample size (or sliding window size) for each $x_i$ sensor.

One option is to take a larger fixed size sliding window over your time series, $X^{(t)} \in \mathbb{R}^{W \times N}$ where $W$ is the window size.

Then, there are methods that perform regularization for all $x_i$ such as group lasso or group elastic net or ridge regression. The subsequence for a sensor is one group.

In effect, this adjusts the $b_i^{(w)}$ for all $x_i^{(w)}$ giving a specific weight to each one of the subsequence points in your window, effectively adjusting the window size or sample size separately per $x_i$.

If I understand correctly, you wish to use only specific sample values of $$X_i^{j}$$ ($$i$$ for the time period and $$j$$ for the predictor index) to predict $$Y_i$$.

Say you have the sampler values: $$S_i^j = 1$$ if predictor $$j$$ is selected in time period $$i$$. Then you could simply run the following regression:

$$Y_i = \beta_1 S_i^1 X_i^1 + \beta_2 S_i^2 X_i^2 + ...$$

and this would only fit to the regressors on the sampled time periods.

p.s. From your explanation it didn't seem to me that there was anything inherently specific to time series...