9
$\begingroup$

Main question up front: what are differences between econometrics/social science statistics that and industrial statistics that people switching between the two should be are of?

I got a PhD in mathematical statistics in December and have a job now that is in a different area of statistics than I studied and cared about for years. I studied econometrics for years but now I work in operations research and use industrial statistics. In particular, most of the studies I see now involve designed experiments; this is almost never the case in econometrics.

As my job is to be a consultant, I feel bad whenever I make wrong claims or bad advice to people who are not experts in what I know, and I think I've now twice made a wrong statement. Regarding whether to drop terms from statistical models with insignificant p-values or whether one should favor including or excluding terms that border on being relevant, I tapped on my econometrics lessons and said: the cost of including an irrelevant term is larger standard errors but the cost of excluding relevant terms is biased parameter estimates, which is worse. This is technically true, but biased parameters are irrelevant in the context of designed experiments. In order for parameters to be biased, regressors need to be correlated with each other; since they cannot be correlated in a designed experiment, omitted variable bias cannot be a problem, or should not be expected to be a problem.

(Aside: I still tend to favor including potentially irrelevant factors, though, and deleting parameters from models based on large p-values or selecting parameters based on aggressive AIC optimization makes me very nervous; I would rather adopt a procedure that accounts for model selection automatically, such as LASSO regression, or use some other basis for deleting parameters other than p-values, such as looking at AIC for hand-selected models or looking at Normal plots, but model selection is still what keeps me up at night.)

This is the second time in the span of a few months I offered incorrect advice (the other time hinged on a misunderstanding of terms) and I want to do what I can to minimize this occurrence. My background in econometrics has revealed itself to be a potential stumbling block (sometimes; other times it can be an advantage) and I would like to see a list of other potential major differences between econometrics or social science statistics and industrial statistics that I should be aware of. What else is something I shouldn't say is true because it's a concern in econometrics since it's not a concern in industrial statistics?

$\endgroup$
5
  • 2
    $\begingroup$ Do you mean designed experiments vs observational studies when you say industrial statistics vs econometrics? Stating that clearly might make it easier to answer your question. $\endgroup$
    – Dave
    Jun 26 at 0:12
  • 2
    $\begingroup$ @Dave That's a much better phrasing of the problem than what I said, but highlighting the differences in the fields could still be useful. In this case, I tripped up on designed experiments vs. observation, but in the future, it could be something else that economists worry about that other areas of statistics don't. $\endgroup$
    – cgmil
    Jun 26 at 0:16
  • $\begingroup$ There may be more similarity between LASSO and variable selection based on p-values than you think. If you are sceptical about the latter, you should probably be careful with the former, too. $\endgroup$ Jun 26 at 8:51
  • $\begingroup$ @RichardHardy Care to elaborate, maybe with some links? $\endgroup$
    – cgmil
    Jun 30 at 16:39
  • $\begingroup$ Sorry, I do not remember the source(s) of this. I studied LASSO in a bit of detail some 4-6 years ago and have forgotten a lot by now. $\endgroup$ Jul 1 at 10:58
12
$\begingroup$

Firstly, good on you for reflecting on past errors in your consulting work. You obviously care a lot about giving good advice to your clients, and your self-criticism is likely to make you a good practitioner in the long run. People hiring graduate students as statistical consultants ought to be aware that you get what you pay for, and they are hiring someone who is still in their formal education phase.

Statistical theory ought to be thought of as a single body of knowledge on inference, prediction, etc. While it contains a number of different philosophies, methodologies and models, the whole point of the field is that it abstracts away from particular applied problems, and is therefore applicable to any of the sciences when they involve inference, prediction, etc. Statistical theory is (or at least ought to be) the same across applied fields, but there is certainly a difference in emphasis and applications in different applied fields, due to the different kinds of questions and information constraints. Irrespective of the particular fields you are working in, here are some crucial things to bear in mind.

  • Always consider/scrutinise the sampling method: Some fields have data that is collected through nice sampling methods that accord with simple statistical model assumptions, so it is easy to get complacent if you practice exclusively in a field like this. For consulting work, it is important to remind yourself that you must consider/scrutinise the sampling method, in case it raises issues (e.g., informative sampling problems). In econometric work, many of the variables under study are macroeconomic estimates from large-scale survey work, where the estimates use a lot of underlying data (e.g., data from a census, tax agency, etc.). In the social sciences much of the research is based on smaller-scale surveys or other small-scale sampling methods, and sometimes these do not involve properly randomised sampling. Make sure you always turn your mind to the sampling method when you meet a new statistical problem.

  • Be mindful of the distinction between causal inference versus predictive inference: In predictive inference we only care about the predictive accuracy of a model/method, and we do not care much if an intermediary statistical association is estimated poorly or whether a statistical association is due to a causal relationship or not. In general, these problems are "easy". Contrarily, in causal inference we care about the causal effect of variables and so it then becomes important to consider whether specific parameters in a model are estimated well, and whether any statistical association detected is due to a causal relation or not. Generally this involves having knowledge of experimental theory, particularly the distinction between controlled and uncontrolled experiments. Causal inferences can be made from randomised controlled experiments, and in uncontrolled cases we usually fall back on trying to "control" for unobserved confounding factors using regression methods. Learn as much as you can about experimental methodology, and also learn to judge how well (or badly) uncontrolled experiments can make causal inferences through controlling for confounding variables.

  • Always consider problems of "over-fitting": Your general tendency to include model terms even if they don't pass individual "inclusion tests" (and your general skepticism of stepwise regression) is quite reasonable in my view, and accords well with the wisdom of the profession. Methods like that often lead to "over-fitting", so it is often reasonable to throw in a bunch of model terms and keep them in the model even if some look like they might be irrelevant.

  • Keep an eye on the "big picture": Sometimes in consulting work (particularly in the social sciences) statisticians can agonise over squeezing every last drop of information out of a small sample, and questions of whether to include or exclude a single model term then seem like a big deal. If you find this is happening, it may be a sign that the sample size is too small to give reliable and robust inferences (without a heavy dependence on model choices) and the best advice may be that your client should get more data. Sometimes statisticians are loath to propose more data as the answer to a problem, since the essence of our subject is to make the best possible inferences/predictions from what information we have, but in some cases the sample size is the "big picture" and the inclusion/exclusion of a model term is the "small picture".

$\endgroup$
5
  • 3
    $\begingroup$ Excellent explanation by Ben. One thing I've noticed as far as difference between industrial statistics and econometrics is the issue of parameter instability. I can't say all the time ( and I don't claim much experience in industrial statistics ) but, generally speaking, in a well designed experiment , one would expect the estimates not to change too much if the experiment was repeated over and over again. OTOH, in econometrics, because of the time factor and structural change possibilities, one observes instability of parameter estimates in almost everything that is done. $\endgroup$
    – mlofton
    Jun 26 at 5:18
  • 1
    $\begingroup$ Great answer! One quibble, though. It is interesting that what you call "over-fitting" results in smaller models with less variance and higher bias compared to the baseline of not excluding variables as aggressively. But smaller variance and higher bias are the hallmarks of underfitting. I think I understand what you mean, but this is a curious case of ambiguous terminology. Any thoughts on this? (A related thread is "Can overfitting and underfitting occur simultaneously?".) $\endgroup$ Jun 26 at 9:00
  • $\begingroup$ Yes, I'm not sure of the best terminology here --- I have in mind the situation in which the analyst essentially cherry-picks model terms by removing non-significant terms to give the impression of a lower variance than is actually there. This relates to the "garden of forking paths" that Gelman talks about. Happy to change the terminology in the answer if there is a better term for the problem I have in mind. $\endgroup$
    – Ben
    Jun 26 at 10:18
  • 1
    $\begingroup$ If only I knew a better term, I would suggest it. I like your point, do not get me wrong. Just found something curious about the terminology, which I thought was worth sharing. I was also wondering if perhaps there was some mistake I was making which you could point out. $\endgroup$ Jun 26 at 12:04
  • $\begingroup$ No mistake on your end --- I think you are right that "over-fitting" might not be quite the word for that (though these all seem to be terms of art). If I think of a better term I will edit it. $\endgroup$
    – Ben
    Jun 27 at 0:40
1
$\begingroup$

I assume by "industrial" statistics you mean 'applied' statistics, where the theory behind it is not as relevant. But as the above reply says, all fields of statistics SHOULD be more-or-less the same, although some emphasize different things than others.

My own work is kind of in-between academic, social-science-type of statistics and applied industrial statistics. I do a lot of data exploration (biomedical data) and often end up doing regressions and things in an attempt to both explain and predict outcomes. My advice is do indeed include variables in your regression models that may not be statistically significant IF they improve model fit (or if taking them out reduces model fit). Two little diagnostics I make use of (talking about linear regression here) is the R-squared value and the standard error of the regression. The R-squared shows how much the predictors explain the outcome variable, i.e., how well one accounts for the other. If you keep adding variables into your model, the R-squared ALWAYS goes up, even if those variables have no real value and don't predict things any better. The standard error of the regression is the average deviation from the regression line that your outcome variables have; i.e., how "tight" the points are around the line. Both are often overlooked in regression, and both are very useful.

Some other tips of the trade:

  1. Sample size matters a lot; often a result won't be significant NOT because of the effect size, but because of the sample size. Become familiar with statistical power (to detect a significant result) and use it in your reports as necessary to larger sample sizes in the future and to qualify your results.
  2. Graph your results. And intermediate steps. A picture is worth a thousand words. And that includes getting skilled with graphing in your software.
  3. Learn new techniques often. There are free and paid forums and statistical learning sites on-line. Use them. As Julia Child said, "learn every time you cook." And in statistics, you are cooking numbers! Scan the literature and see how others are presenting their results; you'll get good ideas for data presentation at least.
  4. Keep improving your writing. The write-up of your results is vital. People hear what they want to hear sometimes, so learn to phrase things carefully, especially any caveats or assumptions or violations of assumptions about the statistical models you use (which isn't always a deal-breaker).
    4a) Publish something once in a while, even if it's just a minor technical finding or example in your field with a colleague as co-author. It boosts your credibility and is helpful to get your next job. And keep your resume updated at all times.
  5. Keep it simple. The simplest solutions (for example in regression) are often the best for practical utility.
  6. Learn bootstrapping. I've just become familiar with it and it is intriguing. It may be quite useful for small samples.
  7. Learn TWO statistical software packages. I don't know what field you're in exactly, but I recommend R and SAS as one of them. (I'm an SPSS man myself, but I could probably earn literally twice as much money doing the same kind of work if I had learned SAS instead. They don't teach you those lessons in graduate school).
$\endgroup$
1
  • 2
    $\begingroup$ (1) Bootstrapping doesn't work on small samples: you can't manufacture information by resampling. (2) Although the "cooking numbers" simile is understandable, it is unfortunate due to its similarity to pejorative idioms like "cooking the books." $\endgroup$
    – whuber
    Jun 29 at 18:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.