1
$\begingroup$

I researched the topic on the internet and to the best of my knowledge there is no existing literature on the subject comparing and contrasting the pros and cons of the two methods. It seems that feature selection solves the same problem as instrumental variables (reduction in bias) but possibly in a more automated and efficient manner. Is this correct, or can they be used simultaneously with each other? Can you provide examples of such a reason?

$\endgroup$
  • 2
    $\begingroup$ I don't know what feature extraction is, but IV is a method for causal inference. It is not unbiased, but rather consistent as N gets very large. Are you interested in prediction, or causality? if the latter, you're not interested in trading variance for bias. $\endgroup$ – generic_user Nov 3 '15 at 2:21
  • $\begingroup$ Feature extraction is basically going through data to gather additional features which can explain the variation in what you are trying to predict. For example, deep learning can be used to gather features for input into a regression mode. I am interested in both prediction and causality. Could you elaborate on how you approach the problem depending upon either task? $\endgroup$ – Hidden Markov Model Nov 3 '15 at 2:42
4
$\begingroup$

I would strongly caution against conflating these methodologies. There is no existing literature on comparing instrumental variables and feature extraction because they accomplish two very different things, and using one to perform the task of the other will result in garbage analysis.

Instrumental variables is a technique used to correct for bias in coefficient estimation most commonly caused by simultaneity, measurement error, or omitted variable bias. These types of biases are often umbrellaed under the term endogeneity.

Feature extraction, and related techniques are useful for selecting the best combination of variables/features for predicting a response out-of-sample (sorry if my terminology is hard to follow, it's a mix between data science and econometrics). This would seem to lend itself to reduction in omitted variable bias, and so one may be tempted to conflate instrumental variables and feature extraction here.

To the extant that both instrumental variables and feature extraction correct for omitted variable bias the difference is this:

Instrumental variables aims to correct for bias in coefficient estimation, so that the effects of the features on the response (econometric terms: the effects of the explanatory variables on the outcome) hold causal interpretations

Feature selection aims to, among other things, correct for bias in prediction, there by improving predictive accuracy.

Instrumental variable methods are not designed to improve predictive accuracy, in fact techniques which utilize instrumental variables will always predict worse in-sample than their endogenous counterparts and this poor prediction will likely carry over out-of-sample. For the case of least squares, this is made evident by the Gauss-Markov theorem, instrumental variables (specifically two stage least squares) is not BLUE because it is inefficient relative to ordinary least squares.

At the same time, feature extraction doesn't necessarily facilitate a causal interpretation between features and response. Though it is true that adding a lot of variables can reduce the chance of endogeneity through omitted variable bias, not all endogeneity is caused by omitted variables (see the example below with simultaneity) and even in the case that it is, the combination which offers the best predictive accuracy is not guaranteed to have a causal interpretation (rooster crows then sun rises, but the rooster does not cause the sun to rise).

Furthermore, some omitted variable problems are caused by the existence of important features that cannot be observed. A classic example; what is the effect of college education on future wages? If you run a simple regression you will find the two are positively correlated, but a person's motivation effects both their level of college education and future wage. There is no direct measure of someones motivation so it is an omitted variable. In this context it is challenging to justify causal interpretations by adding a plethora of other observable features.

An example with Simultaneity:

A rise in the price of oil causes oil companies to produce more oil, but at the same time an increase in production of oil causes price of crude oil to fall. Although, simply regressing price on production or vice versa will give you a (partial) correlation coefficient, that coefficient will be a bias estimate of causality (probably bias toward zero), since the direction and structure of the causality is not well defined.

Note this is not an omitted variables problem. No matter how many variables I add or fancy feature selection technique I use this causality problem would persist (One possible exception being a difference in difference regression design, but that's not really what people do in feature extraction).

The bottom line is price and production are correlated, so you can still use one to predict the other, in some contexts quit well, without knowing anything about the causality structure. However, if you are interested in the causality structure (i.e. if you work for the oil company) you need to find a way to shock the system. In the case of regressing price on production, you need something that will shock the production process but will not directly effect market price after taking production and other variables in the regression into account. That something is an instrumental variable (an example here: an index measuring middle-eastern conflict ).

Unlike feature selection you would not stick the instrument (or any function of it) in the predictive model directly. Note that I was very careful to write that the instrument has no effect on the response after taking the effect of the already included variables into account. Thus by the very definition of a valid instrument, there would be no benefit to including it in the model.

$\endgroup$
2
$\begingroup$

Instrumental variables (IV) are used to deal with endogeneity, which undermines unbiasedness and consistency issues. For instance, a stock example in econometrics is ouput and price equations: $$Q_D=\alpha_0+\alpha_1 P+\alpha_2 I+\varepsilon_D$$ $$Q_S=\beta_0+\beta_1 P+\beta_2 InputPrice+\varepsilon_S$$ $$Q_D=Q_S$$ The first equation is demand. If you try estimating its parameters, you'll run into an issue with a simultaneous equations. You can solve for price $P$ and see that it's correlated with $\varepsilon_D$, hence the OLS regression will not be consistent in estimation of the coefficients of price and income $I$ on the output. Feature selection will not address this issue.

$\endgroup$
  • $\begingroup$ Is it possible that the error/disturbance term in your supply regression is intended to have subscript "S"? $\endgroup$ – Silverfish Nov 4 '16 at 22:22
  • $\begingroup$ @Silverfish, copy/paste typo, thanks for noticing $\endgroup$ – Aksakal Nov 5 '16 at 0:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.