3
$\begingroup$

In my experience it seems me that the interpretation about regression, its meaning and its scope, are debatable and great confusion exist about those things. It seems me that confusions are not go away yet. I started to study regression by Econometrics side and I started to be puzzled, some years ago, thinking about regression and its causal role. My perplexity became both, confirmed and go deeper, when I read: Regression and Causation: A Critical Examination of Six Econometrics Textbooks - Chen and Pearl (2013). Great light was done to me by this article: Shmueli - To explain or to predict - Statistical Science 2010, Vol. 25, No. 3, 289–310. In last article the Author use the word explanation as synonym of causation. To now I’m feel myself more confident about the distinction between prediction and causation. I asked and replied in some related post in this site, as example:

Endogeneity in forecasting

Regression and causality in econometrics

However in his article Shmueli spent some word about description. Unfortunately those word are very few and the only clear thing that we can understand from those are: description is different from both causation and prediction; the role of description are much less important than the others two.

My main question is precisely: what description meaning is? What are the main distinction from description and the other two role for regression? Can you give me a simple as possible example where those differences emerge clearly?

Before to reply, to read the follow seem me needed.

I have found in the web a set of recent slide where Shmueli underscore more clearly that the role of description exist and that is different from others two. To Explain To Predict or To Describe? – satellite conference August 15-16 2019 Kuala Lumpur (Malaysia). You can easily download those. These slide speak precisely about the confusion that again exist in both Academia and Industry about the role of regression. These slide are good. However, unfortunately, too few words are spent about description yet.

I understand that description is markedly different from prediction because, first of all, overfitting and related out of sample measures are the core for prediction while become irrelevant for description scope. However it seem me that dangerous conflating area remain between causation and description. Infact, let me note that a couple of slides was insert for underscore the core difference between: prediction-causation and prediction-description but no one for causation-description.

In my mind I justify, but not appreciate much, the existence of description so: given the variables of interest I estimate regressions in order to reveal pure statistical dependencies. This can be a reply for my question above. This my related question can help (Joint probability distributions and regression)

However, unfortunately yet, at slide 23 Shmueli says something that contradict my position. Because she say that significance (difference from 0) are relevant for the variable to “remain” in the model. Why? It seems me that no dependencies is a respectable descriptive result. Worse, if there are variables selection issue also for description purpose this selection looking for what? It seems me that in this case conflations, mainly toward causation side, hardly will go away. My mistake?

EDIT: Often the regression and in particular the linear regression, that represent the object on which we are most focused here, are presented without the distinctions above or at least them do not emerge clearly. More precisely several presentations leave the impression that once reached a sort of “best specification” (no bias) the regression is ready for any scope and it represent, simultaneously, the best for each of these. All the above precisely contrast this perspective. Regression can be useful for each of three scope above but it must been builded keeping in mind only one. Said that, the differences above come from the scope but the algebra of linear regression remain the same, then the core problem for one scope can play some less important role in any others. This question is strongly related Omitted variable bias vs. Multicollinearity. Moreover, we have to remember that all scope above are though for some kind of “induction/inference”, more data are always better. Basically we observe one sample and would to say something about population or at least something that are not observed yet.

Just here let me note that “regression for description” seems me a misnomer it seems refer on “descriptive statistics”t; instead it is focused on inference about parameters. “Regression for associations” seems me much better as name.

About causation the endogeneity is the main problem, about prediction it is overfitting. At this stage I do not understand clearly if for description particular issue exist. Shmueli suggest multicollinearity and significance of regressors. About the former I’m only partially convinced (see below), about the last I’m almost convinced that is not. From comments was suggested that overfitting, defined as in it, can be problematic for any scope. I'm not convinced about this point also. My motivations follows. They are related to linear regression. More in general the distributional problems and functional form for regression matters but those problem are out of my scope here. Said that:

why endogeneity is not an issue for description? Is true that both are focused on parameters inference. However in causation the concept of bias is related to causal interpretation of at least one parameter while in description the concept of bias not matters. Parameters are unbiased per se in the sense that, stated the existence of conditional expectation and the invertibility in design matrix, parameters are given (as the covariates) and them genuine associational meaning always remain.

Why overfitting is not an issue for description? In linear context overfitting can appear only if we add too many regressors, but we already said that them are given. If we have many regressors and too few data is possible that $R^2$ is much higher than is asymptotic counterpart (overfitting). Similarly is possible that parameters estimators are very imprecise. However we can do nothing. If regressors of interest are given, then, also the best linear approximation of conditional expectation, by linear regression estimated with OLS, are given.

Why (near) multicollinearity only partially is an issue for description? For the same previous motivations. Strongly linear correlation among some regressors are a bad news for inference that is the scope. However we can do nothing if not to separate the collinear regressors and perform several regressions that not suffer from collinearity. Unfortunately in this case the goal is only partially or, at most, indirectly achieved.

$\endgroup$
9
  • 1
    $\begingroup$ It is Shmueli, not Shamueli. I wonder why you say overfitting is irrelevant for description. (This is a tough question as both overfitting and description need to be carefully defined first.) I read somewhere on CV that description in the statistical sense is essentially summarizing the probabilistic (thus not necessarily causal) relations between variables. This agrees with the position you express in the penultimate paragraph. $\endgroup$ May 3 '20 at 18:43
  • 1
    $\begingroup$ Also, what do you mean by In the accepted answer the concept of OVB, surely relevant for causation, is used for description. There is a conflation between them?? Which part of my answer are you pointing to specifically? I address both description and causal inference in the answer. I discuss OVB in the context of causal inference, but not (in any detail) in the context of description (though I imply OVB matters for description). So I wonder where you see the possible conflation. $\endgroup$ May 3 '20 at 18:45
  • $\begingroup$ First of all thanks for comments. About overfitting: I looking for an exhaustive explanation about descriptive role for regression, then, I can make mistakes. However starting from the few information that I have and stated that the three scope above are different, it seem me that overfitting is out of scope for description. This primarily because out of sample statistics are never mention for description; just terminologically this seems me consistent. I intend overfitting only as relation between, in vs out measures. $\endgroup$
    – markowitz
    May 5 '20 at 7:32
  • 1
    $\begingroup$ Then, also only terminologically “omitted” become a nonsense. More important, causation and description are both focused on inference about parameters. Now, OVB probably is the main problem in causation. Then, it seems me that if we admit OVB also for description the conflation between them become inescapable. Finally, the material that I have is very unclear about difference between causation and description. $\endgroup$
    – markowitz
    May 5 '20 at 7:34
  • 1
    $\begingroup$ In linear context overfitting can appear only if we add too many regressors, but we already said that them are given. An interesting argument. I started writing a comment to argue you are mistaken, but now I think I agree with you... Maybe a bit later I will understand more and change my mind again, but for now this looks sensible. Regarding the rest of your edit/update, I do not have anything strictly against what you are saying, at least in terms of concrete items. $\endgroup$ May 8 '20 at 17:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.