12
$\begingroup$

I'm fairly new to statistics. I'm not clear on the meaning of independent and dependent variables and the relationship to what's being predicted.

In my text, as an example there is a data set containing many instances of the following:

  • a person's salary

  • a person's age

  • the year they earned that salary

  • their education level

The book mentions trying to predict their salary from the other three variables. Does this mean the other three are the independent variables and salary is the dependent variable?

When this data is arranged in a spreadsheet, with rows being people and columns being variables, something interesting appears. There is symmetry between all the variables. None of them holds a special place in the spreadsheet, they each have their own column.

Which then leads me to ask, could we pick another one, say age, and predict that from salary/year/education? Is age now the dependent variable?

In high school statistics I learned that independent variables have some degree of independence... say the weather is independent from what I have for dinner. There's not much effect that one has on the other.

But in statistics, can the independent variables be regarded as the "things we are using to make the prediction," while the dependent variable is the "thing being predicted?" Is there still a need for independent variables to really be independent in a real-world sense?

$\endgroup$
  • 1
    $\begingroup$ For me, at least, weather and dinner are definitely not independent variables - I'm not going to make a hearty stew if it's 95F and humid, nor will I be grilling burgers in the middle of a thunderstorm, and the likelihood of soup is inversely correlated with temperature! $\endgroup$ – Nuclear Wang Oct 24 '19 at 13:20
  • 3
    $\begingroup$ Some regression textbooks are sensitive to this confusion about the use of "independent" and elect to use other terms. I like "explanatory variables," but have also seen "covariates." The ML community seems to like "features." $\endgroup$ – whuber Oct 24 '19 at 17:11
  • $\begingroup$ @NuclearWang, point taken but it was just the first dumb example I could think of that involved real-world factors. I didn't want to say that two card draws with replacement are independent; I wanted to bring in something real-world. And most real things affect each other, possibly. How about "the weather today and my wife's age." $\endgroup$ – composerMike Oct 25 '19 at 18:24
  • $\begingroup$ Not a very good example: the passage of time affects your wife’s age certainly and the weather through climatic change broadly. Most spurious correlations arise through variables changing with time in different ways! Dealing cards and tossing coins are hackneyed examples precisely because weak dependence is so pervasive in phenomena in which independence does not arise on purpose. $\endgroup$ – Nick Cox Oct 25 '19 at 19:13
  • $\begingroup$ @NickCox, that's what I was trying to say, that most real world things affect each other in some way. Assuming we pick a random person, don't repeat the experiment and ask "what is your age and what is the weather" that's pretty close to independent right? Does passage of time have to be considered even then or is there a need for an experiment to be repeatable? $\endgroup$ – composerMike Oct 25 '19 at 20:41
13
$\begingroup$

The questions "What do you want to predict?" and "What is the outcome or result here?" often have the same answer, but not always.

The terminology of independent variables is widely considered overloaded in statistical sciences. Numerous writers and researchers -- over at least the last several decades -- have suggested using other terms, although there is little consensus on what the best terms are. Some terms are predictors, explanatory variables, controlling variables, regressors, covariates, inputs, ....

The term dependent variable similarly is often substituted with something more evocative. For some time response seemed to lead the field of alternatives, but outcome and output have been among frequent recent terms. I note without enthusiasm the existence of regressand.

DV and IV are common abbreviations in some fields, sometimes seeming to tag initiates engaged by mutual consent in regression rituals. An objection to DV is that Deo volente remains a standard expansion for many people. A bigger objection to IV is that it is bespoke (by many economists in particular) for instrumental variable.

Still, the old terms linger on, and my impression (no names here) is that they are still often recommended in textbooks which on other grounds I regard as poor or incompetent.

Terminology aside: There is no absolute implication that so-called independent variables in a regression are statistically independent of each other, and indeed that fact is one of several objections to the terminology.

There are even situations in which predictors are deliberately introduced that are highly correlated with each other. Fitting a quadratic in $X$ and $X^2$ is a case in point, as $X$ and $X^2$ are not mutually independent. It's, however, foolish to include two predictors with essentially the same message, as say Fahrenheit and Celsius temperatures. In practice, good software has traps to detect that situation and drop predictors as needed, but the researcher still needs to be careful and thoughtful about their choice of predictors. The ideal -- easier to advise as a principle than to ensure in practice -- is for predictors to have a clear rationale and to use no more predictors than are needed for the purpose, and that are reasonable given the size of the dataset.

Your example is instructive. Usually salary depends on age, sometimes directly if an individual moves up a salary scale, but more often indirectly through salary being affected by promotion or moves to a different job and those being affected by greater experience, expertise, reputation, and so forth. Conversely, sometimes older people are less attractive to employ (e.g. sports people past their peak). But the crux is that a salary raise doesn’t affect age, whereas a change in age may affect salary (on average, which is what we care about here). Causal paths can exist in indirect ways.

All that said, in different problems age is unknown and the goal is to predict it. This is standard in archaeology, forensic sciences, and several Earth and environmental sciences.

| cite | improve this answer | |
$\endgroup$
9
$\begingroup$

@NickCox gave an excellent answer. A couple additions:

You ask

But in statistics, can the independent variables be regarded as the "things we are using to make the prediction," while the dependent variable is the "thing being predicted?"

To give an explicit answer: Yes, that is often how the terms are used. I use them that way, myself.

Second, the preferred terms seem to vary by field as well as by individual. My PhD is in psychometrics (in the psychology department) and "independent" is very common there.

Third, the meaning of other terms on Nick's list also varies. Some people use "covariate" to mean "all the X variables" while others use covariate to mean the nuisance parameters that you aren't really interested in but have to account for.

Finally, other terms have their own issues: "Predictors" - sometimes we aren't really interested in predicting. "Explanatory variables" - similarly, we sometimes aren't interested in explanation (and, sometimes, we are interested in both explanation and prediction). "Regressor" isn't bad, but it sort of implies that we are doing some form of regression, but then there are independent variables in methods that are not called "regression".

It's a mess!

| cite | improve this answer | |
$\endgroup$
  • 2
    $\begingroup$ Good answer (+1). While "regression" is sometimes used to refer to Gaussian models, it is worth noting that the term can also refer broadly to any problem in which you seek to find the conditional distribution of a "thing being predicted" conditional on "things we use to make the predictions". My understanding is that there is a broad sense of "regression" that would encompass GLMs, GLMMs, etc. $\endgroup$ – Ben Oct 24 '19 at 12:32
  • 2
    $\begingroup$ @Ben, there is the division of supervised machine learning into regression and classification, that could be relevant w.r.t. your last sentence. $\endgroup$ – Richard Hardy Oct 25 '19 at 12:44
4
$\begingroup$

As you have correctly noticed, the term 'independent' has completely different meanings depending on context.

Statistical independence is what you are describing between the weather and your dinner. These two events are independent in the sense that the value of one does not affect the other. There are more formal mathematical definitions of this independence, but your basic understanding is right.

Independent variables in regression is a term that refers to the set of $x$ variables. Sometimes they are also called predictors or covariates. Indeed, as you mentioned in your example, you can pick age as the response (the dependent variable) and the other three as your independent variables. However, whether this is a good idea or not depends on the practical purpose of what you are doing. In reality, you are interested in predicting salary based on other variables, so you pick salary as the dependent variable and call the others independent variables. There is nothing that forces you to call one of them the dependent variable beforehand - it's entirely up to you and depends on the question you are trying to answer.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.