I have a large panel-like data set: about 15,000 individuals, on average 350 time points, two dozen variables (plus some more variables we left out for context-specific reasons). What I want to achieve is a model explaining one dependent variable $y$ by (some of) the other (i.e. explanatory) variables $x$. As a first step, this should be something like a linear regression model.

Properties of the data:

  • $y$ follows a non stationary process (not rejected by Dickey–Fuller test) and increases linearly over time on average.
  • One of the explanatory variables $x_1$, with $x_1(t) \geq x_1(t-1)$, is way stronger correlated with $y$ than any other variable.
  • Most of the variables do not follow a Gaussian distribution, some (including $y$ and $y^\prime$) have a heavy tailed distribution.
  • Some of the variables are strongly correlated.

What I've done/calculated so far:

  1. Pearson correlation of $y$ with each $x_k$ separately.
  2. Pearson correlation of $y$ with each $x_k$, controlling for $x_1$.
  3. Sampling only every $n^{th}$ time point ($n$ chosen so that the autocorrelation of $y$ has decayed below 1/e), and doing simple linear regression on the sample, using time as just one more variable. Regression of $y$ on $x_1$ yields $R^2=0.36$; including all variables yields $R^2=0.40$.
  4. Taking $y^\prime$, i.e. $y(t+1)-y(t)$ as dependent variable. The autocorrelation of $y^\prime$ decays really fast. Regression of $y^\prime$ on $x_1^\prime$ yields $R^2=1.5\cdot 10^{-4}$, regression of $y^\prime$ on all $x_k$ yields $R^2=2\cdot 10^{-4}$.
  5. Taking $\log(y(t+1)/y(t))$ ('log return') as dependent variable. Regression on all $x_k$ yields $R^2=0.003$; regression on $\log(x_1(t+1)/x_1(t))$ yields $R^2=0.0012$.

Despite the low $R^2$, the F-test yields a very low p-value and most p-values for the coefficients are very low. All regressions include a constant term.

What I'm looking for:

  • In the best case a (pointer to a) 'standard' way to derive a model from data of this size (and kind), and a quantification of how much sense the resulting model makes.
  • Criteria which of the explanatory variables to use in a model (the size of the data shows even really small effects significantly).

closed as too broad by Michael Chernick, gung Aug 14 '18 at 21:04

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • $\begingroup$ What happens if you regress the log-return $y$ on log-return $x_1$? Or does that not make sense for this data? $\endgroup$ – Hong Ooi Jul 31 '13 at 18:49
  • $\begingroup$ Doesn't make too much sense, I guess; did it anyway for curiosity. Had an error in 4., see update. $\endgroup$ – B Fuchs Aug 1 '13 at 18:44