9
$\begingroup$

Suppose I have longitudinal data of the form $\mathbf Y = (Y_1, \ldots, Y_J) \sim \mathcal N(\mu, \Sigma)$ (I have multiple observations, this is just the form of a single one). I'm interested in restrictions on $\Sigma$. An unrestricted $\Sigma$ is equivalent to taking $$ Y_j = \alpha_j + \sum_{\ell = 1} ^ {j - 1} \phi_{\ell j} Y_{j-\ell} + \varepsilon_j $$ with $\varepsilon_j \sim N(0, \sigma_j)$.

This is typically not done since it requires estimating $O(J^2)$ covariance parameters. A model is "lag-$k$" if we take $$ Y_j = \alpha_j + \sum_{\ell = 1} ^ k \phi_{\ell j} Y_{j - \ell} + \varepsilon_j, $$ i.e. we only use the preceding $k$ terms to predict $Y_j$ from the history.

What I'd really like to do is use some kind of shrinkage idea to zero out some of the $\phi_{\ell j}$, like the LASSO. But the thing is, I also would like the method I use to prefer models which are lag-$k$ for some $k$; I'd like to penalize higher order lags more than lower order lags. I think this is something we would particularly like to do given that the predictors are highly correlated.

An additional issue is that if (say) $\phi_{35}$ is shrunk to $0$ I would also like it if $\phi_{36}$ is shrunk to $0$, i.e. the same lag is used in all of the conditional distributions.

I could speculate on this, but I don't want to reinvent the wheel. Is there any LASSO techniques designed to get at this sort of problem? Am I better off just doing something else entirely, like stepwise inclusion of lag orders? Since my model space is small, I could even use an $L_0$ penalty on this problem I guess?

$\endgroup$
2
$\begingroup$

You can do cross validation repeatedly from k = 0 to whatever the maximum is, and plot the performance against k. Since the model is being tested on data it hasn't seen before, there is no guarantee the complex models will perform better, and indeed you should see a degradation in performance if the model becomes too complex due to overfitting. Personally I think this is safer and easier to justify than having an arbitrary penalty factor, but your mileage may vary.

I also don't really follow how ordered Lasso answers the question. It seems too restrictive, it is completely forcing the ordering of the coefficients. Whereas the original question may end up for some data having a solution where $\phi_{lj}$ is not strictly decreasing with l.

$\endgroup$
  • $\begingroup$ To add LaTeX to your question, enclose the expression between dollar signs ($). $\endgroup$ – Patrick Coulombe Aug 3 '14 at 1:09
  • 1
    $\begingroup$ (1) From the model alone, it isn't obvious that the coefficient ordering is desirable, but substantively it is reasonable. In a repeated measures clinical trial, for example, there is no substantive reason to expect that a small perturbation of $Y_{j-2}$ to stochastically influence $Y_j$ more than a small perturbation of $Y_{j-1}$. The ordered LASSO makes better use of this a priori knowledge, with the minor risk that it might be not be true. $\endgroup$ – guy Aug 3 '14 at 3:29
  • $\begingroup$ (2) Generally, I wouldn't use this CV strategy at least partially because it is too dogmatic. I can get better predictions by judiciously shrinking a lag, rather than throwing it out entirely. $\endgroup$ – guy Aug 3 '14 at 3:29
  • $\begingroup$ Nir, a useful comment on the ordered LASSO. I've edited my answer to be a bit more comprehensive. Thanks! $\endgroup$ – Sean Easter Aug 3 '14 at 15:54
  • $\begingroup$ Thanks Sean. Guy, I don't think it is too dogmatic. You are not setting a k in stone, but rather allowing it to vary. The k it chooses will be at the onset of overfitting. I also strongly disagree with your statement of supposed a priori knowledge. Something seeming reasonable and knowing that thing are completely different. I must admit there seems to be a resistance in traditional stats to cross validation I've never understood. I'd choose predictive efficiency on out of sample data over adding assumptions any day. $\endgroup$ – Nir Friedman Aug 3 '14 at 17:12
2
$\begingroup$

The ordered LASSO seems to be what you're looking for: It computes the regularized regression coefficients $\beta_{1...j}$ as in the standard LASSO, but subject to the additional constraint that $|\beta_1| \geq |\beta_2|...\geq|\beta_j|$.

This accomplishes the second goal of zeroing out coefficients for higher-order lags, but is more restrictive than the sole restriction of preferring a lower lag model. And as others point out, this is a heavy restriction that can be very difficult to justify.

Having dispensed with the caveats, the paper presents the method's results on both real and simulated time series data, and details algorithms to find the coefficients. The conclusion mentions an R package, but the paper is rather recent and a search on CRAN for "ordered LASSO" comes up empty, so I suspect the package is still in development.

The paper also offers a generalized approach in which two regularization parameters "encourage near-monotonicity." (See p. 6.) In other words, one should be able to tune the parameters to allow for a relaxed ordering. Sadly, neither examples nor comparisons of the relaxed method are provided. But, the authors write that implementing this change is a simple matter of replacing one algorithm with another, so one hopes it will be part of the coming R package.

$\endgroup$
  • $\begingroup$ Thanks, that's really interesting that this is a recent idea. I actually came up with the same idea discussing the problem with a friend back when I asked the question 9 months ago, but never investigated it in depth! I just assumed that the idea wasn't that novel, or that someone else had written a paper about it already. $\endgroup$ – guy Aug 1 '14 at 22:14
  • $\begingroup$ Quite welcome! I was surprised that it was so recent myself. $\endgroup$ – Sean Easter Aug 2 '14 at 1:03
1
$\begingroup$

The nested LASSO penalty (pdf) could be employed but there are no R packages for it.

$\endgroup$
  • 1
    $\begingroup$ At present, this is more of a comment than an answer. Can you expand it a little, perhaps by discussing the nested LASSO penalty, etc? $\endgroup$ – gung Aug 11 '14 at 18:52
0
$\begingroup$

I know you wrote it as a premise, but I would not use the ordered LASSO without being absolutely sure that this is thing that is needed, because the assumptions of the ordered LASSO are not directly appropriate for time-series prediction. As a counter-example, consider the case where you have a delay-time of, say, ten time-steps between measurement and target. Obviously, the ordered LASSO constraints cannot handle such a effects without attributing nonsense to the first nine parameters.

In contrast, I would rather stick to the normal LASSO and include all previous observation -- particularly because you wrote your model space is small, and coordinate-descent optimization routines for the LASSO (as described here) are working efficiently also for large datasets. Then compute the path for the regularization strength parameter $\lambda$ and look which parameters get included as you go from large $\lambda$ to $\lambda=0$. Especially those included earlier are the important ones.

Finally, you have to choose an appropriate criterion and optimize the parameter $\lambda$ using cross-validation, standard one-dimensional minimization or whatever. The criterion can for example be something as "prediction error + number of included variables" (--AIC criterion-like).

$\endgroup$
  • $\begingroup$ I would obviously not be interested in constraints on the order of the coefficients if I did not have strong a priori reasons to believe it. For the models that I suspect to be likely, heuristically the ordered LASSO should be more efficient. Having a lag-10 coefficient with the other 9 being 0 makes no sense in my substantive setting. This is a problem that my colleagues have worked on (ordered based shrinkage on the lags), but they used Bayesuan ideas and so wouldn't consider a (non-Bayesian) LASSO. $\endgroup$ – guy Aug 3 '14 at 15:13
  • $\begingroup$ Ok, you seem to know what you do. But remember that the ordered LASSO is more strongly constrained than your "once zero - always zero" statement. Alternatively, you could also consider a model where the parameters enter in a multiplicative way. Then, the relative importance may either grow or decrease until a coefficient becomes zero. $\endgroup$ – davidhigh Aug 4 '14 at 7:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.