3
$\begingroup$

It's sometimes said (e.g. in this talk) that doing univariate analysis before multiple regression may lead to kicking off useful features and other mistakes. So, my questions are as follows.

  1. Are there any simple example models showing that such an effect may occur?
  2. If so, does it mean I need not to make univariate analysis at all? Or it just means that I can't reject a feature only due to the bad result of a univariate test?

EDIT: by univariate analysis I mean feature selection

Improve this question
$\endgroup$
4
  • $\begingroup$ Can you clarify your question? What kind of univariate analysis are you taking about (seems to me you refer to ANOVA)? What kind of variables are you analyzing (continuous or categorical). Also, note that to me and many others, multiple regression analysis IS an univariate test (univariate = 1 response variable, multivariate = 2 or more response variables). In this case, there's no issue to do an ANOVA and then run a multiple linear regression. Actually I've seen this often but they answered different questions. $\endgroup$
    – Mud Warrior
    Commented Oct 11, 2016 at 12:53
  • $\begingroup$ Do you mean a univariate (simple) regression of the form $y_{t}=\beta_{0}+\beta_{1}x_{1}+\epsilon_{t}$ vs, a multiple regression, $y_{t}=\beta_{0}+\beta_{1}x_{1}+\beta_{2}x_{2}+\dots+\beta_{k}x_{k}+\epsilon_{t}$? Or do you mean univariate vs. multivariate time series models, i.e. AR(2) vs. a VAR(2)? $\endgroup$
    – Plissken
    Commented Oct 11, 2016 at 12:54
  • $\begingroup$ If I understand it correctly and you are referring to the former of the two I mention above then no. You need to do a multivariate regression. I can expand on this in an answer if this was in fact what you had in mind. $\endgroup$
    – Plissken
    Commented Oct 11, 2016 at 12:59
  • 1
    $\begingroup$ If you want a very simple model where this is true consider an interaction only model. That is let $X_1$ and $X_2$ be independent binary random variables taking the values $\{-1, 1 \}$ with equal probability, and set $Y = X_1 X_2$. Taken individually there's no relationship between the predictors and $Y$, but jointly they determine $Y$ exactly. $\endgroup$
    – dsaxton
    Commented Oct 11, 2016 at 13:16

1 Answer 1

Reset to default
3
$\begingroup$

In an article available here Sun and colleagues describe what they entitle "Inappropriate use of bivariable analysis to screen risk factors for use in multivariable analysis" which may be of interest to you. When they say bivariable they mean what you refer to as univariate.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.