29
$\begingroup$

I mean some of those variables are strongly correlated among themselves. How / why / in what context do we define them as independent variables?

$\endgroup$
  • $\begingroup$ That's historical and comes from French scientific works. I am trying to find the reference. $\endgroup$ – Alecos Papadopoulos Jul 19 '18 at 11:41
  • $\begingroup$ I would call a set of variables "potentially co-dependent" to avoid inferring the causality. $\endgroup$ – qed Jul 21 '18 at 7:31
  • $\begingroup$ A good question! $\endgroup$ – Rafael Marazuela Jul 21 '18 at 7:39
  • 1
    $\begingroup$ Of interest: Left-hand & right-hand side nomenclature in regression models $\endgroup$ – Alexis Aug 2 '18 at 21:28
29
$\begingroup$

If we pull back from today's emphasis on machine learning and recall how much of statistical analysis was developed for controlled experimental studies, the phrase "independent variables" makes a good deal of sense.

In controlled experimental studies, the choices of a drug and its concentrations, or the choices of a fertilizer and its amounts per acre, are made independently by the investigator. The interest is in how a response variable of interest (e.g., blood pressure, crop yield) depends on these experimental manipulations. Ideally, the characteristics of the independent variables are tightly specified, with essentially no errors in knowing their values. Then standard linear regression, for example, models the differences among values of dependent variables in terms of the values of the independent variables plus residual errors.

The same mathematical formalism used for regression in the context of controlled experimental studies also can be applied to analysis of observed data sets with little to no experimental manipulation, so it's perhaps not surprising that the phrase "independent variables" has carried over to such types of studies. But, as others on this page note, that's probably an unfortunate choice, with "predictors" or "features" more appropriate in such contexts.

$\endgroup$
  • 2
    $\begingroup$ But the choice of the levels of the drug is dependent on what the investigator does which is why I can never remember which is which. $\endgroup$ – mdewey Jul 18 '18 at 12:13
  • $\begingroup$ In machine learning, "features" are often latent, unobserved variables. “Observed features” is more common. $\endgroup$ – Neil G Jul 21 '18 at 18:48
18
$\begingroup$

In many ways, "independent variable" is an unfortunate choice. The variables need not be independent of each other, and of course need not be independent of the dependent variable $Y$. In teaching and in my book Regression Modeling Strategies I use the word predictor. In some situations that word is not strong enough, but it works well on the average. A full description of the role of the $X$ (right hand side) variables in a statistical model might be too long to use each time: the set of variables or measurements upon which the distribution of $Y$ is conditioned. This is another way of saying the set of variables whose distributions we are currently not interested in, but whose values we treat as constants.

$\endgroup$
  • $\begingroup$ So all you are saying that calling input variables as "independent" is wrong practice? @Frank $\endgroup$ – Amarpreet Singh Jul 18 '18 at 11:20
  • 10
    $\begingroup$ They are definitely not assumed to be independent of ANYTHING so it's wrong practice, used only because of habit. $\endgroup$ – Frank Harrell Jul 18 '18 at 12:35
  • 1
    $\begingroup$ "the set of variables or measurements upon which the distribution of Y is conditioned" ... actually I do think of them as (and sometimes call them) the "conditioning variables" or "variables conditioned on", which is not too long a description and works naturally with the notation $E(Y|X)$ $\endgroup$ – Silverfish Jul 19 '18 at 20:44
11
$\begingroup$

I agree with the other answers here that "independent" and "dependent" is poor terminology. As EdM explains, this terminology arose in the context of controlled experiments where the researcher could set the regressors independently of each other. There are many preferable terms that do not have this loaded causal connotation, and in my experience, statisticians tend to prefer the more neutral terms. There are many other terms used here, including the following:

$$\begin{matrix} Y_i & & & x_{i,1},...,x_{i,m} \\ \hline \text{Response} & & & \text{Predictors} \\ \text{Regressand} & & & \text{Regressors} \\ \text{Output variable} & & & \text{Input variables} \\ \text{Predicted variable} & & & \text{Explanatory variables} \\ \end{matrix}$$

Personally, I use the terms explanatory variables, and response variable, since those terms have no connotation of statistical independence or control, etc. (One might argue that 'response' has a causal connotation, but this is a fairly weak connotation, so I have not found it problematic.)

$\endgroup$
  • 1
    $\begingroup$ (+1) I suppose regressor/regressand are the most neutral terms, but I also prefer to explain using explanatory/response. $\endgroup$ – Frans Rodenburg Jul 19 '18 at 8:31
  • 2
    $\begingroup$ I agree with the tendency to prefer neutral terms, but "explanatory" sounds pretty causal to me as in: "The X variables explain why the Y variable acts in the way it does." $\endgroup$ – timwiz Jul 19 '18 at 14:12
  • 1
    $\begingroup$ I take it to mean explanatory in a probabilistic sense -- i.e., it explains changes in the distribution of the response variable. You might be right, but in all these cases the connotation to any causality is weak. $\endgroup$ – Ben Jul 19 '18 at 14:29
  • 2
    $\begingroup$ Explanatory implies causal so is inappropriate. $\endgroup$ – Frank Harrell Jul 19 '18 at 22:43
  • 1
    $\begingroup$ @Frank: I don't necessarily agree with that view. Explanatory is derived from the word "explain" so I take it to imply only that the variables explain the response variable somehow. That explanation could be causal, or it could merely be statistical, and I take it to be the latter. Nevertheless, it does appear that people are interpreting the connotations of these words differently, so I will concede that some will read it as having causal connotations. $\endgroup$ – Ben Jul 19 '18 at 23:59
9
$\begingroup$

To add to Frank Harrell's and Peter Flom's answers:

I agree that calling a variable "independent" or "dependent" is often misleading. But some people still do that. I once heard an answer why:

In regression analysis we have one "special" variable (usually denoted by $Y$) and many "not-so-special" variables ($X$'s) and we want to see how changes in $X$'s affect $Y$. In other words, we want to see how $Y$ depends on $X$'s.

That is why $Y$ is called "dependent". And if one is called "dependent" how would you call another one?

$\endgroup$
  • $\begingroup$ You are saying that Y depends on X's, (so Y is called dependent variable) and by that you mean that X doesn't depend on Y. But there can be cases where X can depend on Y or correlate with Y (so it can't be called "independent" anymore). Any views on this? $\endgroup$ – Amarpreet Singh Jul 18 '18 at 11:34
  • $\begingroup$ No, I don't mean that X doesn't depend on Y. I just mean that the most basic explanation of what regression analysis does is that it describes how Y depend on X. So the most basic name for Y would be "dependent" $\endgroup$ – Łukasz Deryło Jul 18 '18 at 11:36
  • 5
    $\begingroup$ I'm not trying to answer the question "should we call X independent?" but rather "why do we call it independent?", just like in title of your post $\endgroup$ – Łukasz Deryło Jul 18 '18 at 11:38
5
$\begingroup$

"Dependent" and "independent" can be confusing terms. One sense is pseudo-causal or even causal and this is the one that is meant when saying "independent variable" and "dependent variable". We mean that the DV, in some sense, depends on the IV. So, for example, when modeling the relationship of height and weight in adult humans, we say weight is the DV and height is the IV.

This does capture something that "predictor" does not - namely, the direction of the relationship. Height predicts weight, but weight also predicts height. That is, if you were told to guess the height of people and were told their weights, that would be useful.

But we wouldn't say that height depends on weight.

$\endgroup$
  • $\begingroup$ Are you being specific about SEM model? $\endgroup$ – Amarpreet Singh Jul 18 '18 at 11:25
  • $\begingroup$ No. I was thinking of regression. $\endgroup$ – Peter Flom Jul 18 '18 at 11:39
  • $\begingroup$ Ok, so it's just a matter of name. I got confused that calling input variables as "independent" means something. $\endgroup$ – Amarpreet Singh Jul 18 '18 at 11:42
  • 11
    $\begingroup$ DV and IV are common abbreviations (which personally I dislike), but watch out for many economists and some other social scientists for whom IV can only mean instrumental variable. It is less common to encounter people for whom DV can only mean Deo volente (God willing). $\endgroup$ – Nick Cox Jul 18 '18 at 11:50
0
$\begingroup$

Based on the above answers, yes , i agree that this dependent and independent variable are weak terminology. But I can explain the context in which it is being used by many of us. You say that for a general regression problem we have a Output variable, say Y, whose value depends on other input variables, say x1, x2, x3. That is why it is called a "Dependent Variable". And similarly depending upon this context only, and just to differentiate between Output and Input Variable, x1, x2, x3 are termed as independent variable. Because unlike Y it does not depend on any other variable(But yes here we are not talking about there dependency with themselves.)

$\endgroup$
  • $\begingroup$ You answered similar to that of @Ramya R. $\endgroup$ – Amarpreet Singh Jul 20 '18 at 6:40
-2
$\begingroup$

Independent variables are called independent because they do not depend on other variables. For example, consider the house price prediction problem. Assume we have data on house_size, location, and house_price. Here, house_price is determined based on the house_size and location but the location and house_size can vary for different houses.

$\endgroup$
  • 4
    $\begingroup$ Sometimes the so-called "independent" variables in regression are correlated. So they are not necessarily statistically independent. It would be better to call them predictor variables. $\endgroup$ – Michael Chernick Jul 19 '18 at 19:44
  • $\begingroup$ Micheal, Thanks for pointing that out. I have a follow-up question. In cases where we have two predictor variables that are collinear, don't we discard one of them to eliminate the multicollinearity problem so that our predictor variables are independent of each other? $\endgroup$ – Ramya R Jul 23 '18 at 18:56
  • 1
    $\begingroup$ Not necessarily. It depends on whether or not it affects the stability of estimates and how much stronger the prediction is when both variables are included. If two variables have correlation 0.1 they are not independent but the relationship between them is weak. $\endgroup$ – Michael Chernick Jul 23 '18 at 19:26

protected by gung Jul 21 '18 at 11:19

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.