Feature Selection in Multivariate Linear Regression

I have $$m$$ time series and I partition the time series into two sets: set X and set Y. I want to predict set Y using set X. The time series of set Y are $${\bf y}_t$$ and those of the set X are $${\bf x}_t$$. The value of $$m$$ can be large (greater than 100).

To do this I fit a multivariate linear regression model: $${\bf y}_t= \bf a+P {\bf x}_t + \epsilon$$

I can then compute some fitting error (using some method).

I want to know if there exists a way to determine the best way to select the variables that constitute the set X ($${\bf x}_t$$). I want to select only a subset of around 20 variables that can predict the best the other variables using a multivariate linear regression model. We want to select 20 variables to constitute the set X in order to minimize the fitting error.

A reference of existing papers on this would help greatly.

Thanks!

A "Hard" Problem

There does not exist a "best way" to select the variables in your model. If you have $$p$$ possible covariates, you get $$2^p$$ possible models without any interactions or functions of those inputs or lags.

In general, model selection aka feature selection is an unsolved problem and there is evidence that it is an NP-hard problem. (It is especially difficult if you are working with financial time series due to market efficiency.)

We do often use a few heuristic tools, but they are... imperfect at best.

Theory

We always consider theoretical reasons for a variable or function of variables to be in a model. That often yields the greatest insight; however, that requires knowledge of the problem being modeled. We may even keep that variable or function of variables in the model despite it seeming to be insignificant -- just as a control (aka nuisance parameter) since we expect it should matter.

Forward/Backward Selection

We can try forward and backward selection: adding covariates from a base model or subtracting them from a huge model. This is very common but can lead to some problems. This especially runs into difficulty if you do not have complete data for all covariates. In that case, your model will be fit on different datasets depending on what is the maximal dataset with data for all the covariates in the model. (So, add a covariate with only data from the last year and suddenly your model is estimated only on data from the last year.)

Penalized Regressions

We can also look at which coefficients grow fastest as we relax the penalty in a regression which penalizes coefficients. These regressions (typically the LASSO and ridge regression) are biased to try to have a smaller model. However, even this is not guaranteed to be useful.

SVMs and Other ML Techniques

You may hear about people using support vector machines or other ML methods at the problem; however, these have even more flexibility and so can easily lead to overfitting and thus poor feature selection. Basically: if we don't know which covariates to put into a model, adding more parameters or ways to put those into the model doesn't make our task easier. Some industries are now realizing this and have reduced their expectations of ML techniques.

Combining Variables

We could combine variables as done by PCA or ICA; however, neither of those considers the response we are trying to model. There is no guarantee using these will lead to a better model.

Still An Art

The honest truth is this: model selection (aka feature selection) is still very much an art -- and we have little reason to believe this will change soon. Get to know your problem domain well and get to know your data well. That will likely pay off greater dividends than learning some new method.

I recommend you take a look to chapter 3 of Elements of statistical learning book.

There you can see different ways of selecting a subset of your variables, methods that penalize the coefficients for each variable, some of them performing subset selection like LASSO, or methods that use linear combinations of your variables in order to use a reduced number of transformed variables.

It also treats the case of multiple outcome case, as I think it may be in your case.

• Feature selection is a bad idea in general because of its volatility. And it is mistake to say that it "minimizes the fitting error". Consider ridge regression, elastic net, or better, data reduction (unsupervised learning) as a first step. Jul 10, 2021 at 11:35