What are differences between industrial statistics and social science statistics/econometrics that are potential stumbling blocks?

Question

Main question up front: what are differences between econometrics/social science statistics that and industrial statistics that people switching between the two should be are of?

I got a PhD in mathematical statistics in December and have a job now that is in a different area of statistics than I studied and cared about for years. I studied econometrics for years but now I work in operations research and use industrial statistics. In particular, most of the studies I see now involve designed experiments; this is almost never the case in econometrics.

As my job is to be a consultant, I feel bad whenever I make wrong claims or bad advice to people who are not experts in what I know, and I think I've now twice made a wrong statement. Regarding whether to drop terms from statistical models with insignificant p-values or whether one should favor including or excluding terms that border on being relevant, I tapped on my econometrics lessons and said: the cost of including an irrelevant term is larger standard errors but the cost of excluding relevant terms is biased parameter estimates, which is worse. This is technically true, but biased parameters are irrelevant in the context of designed experiments. In order for parameters to be biased, regressors need to be correlated with each other; since they cannot be correlated in a designed experiment, omitted variable bias cannot be a problem, or should not be expected to be a problem.

(Aside: I still tend to favor including potentially irrelevant factors, though, and deleting parameters from models based on large p-values or selecting parameters based on aggressive AIC optimization makes me very nervous; I would rather adopt a procedure that accounts for model selection automatically, such as LASSO regression, or use some other basis for deleting parameters other than p-values, such as looking at AIC for hand-selected models or looking at Normal plots, but model selection is still what keeps me up at night.)

This is the second time in the span of a few months I offered incorrect advice (the other time hinged on a misunderstanding of terms) and I want to do what I can to minimize this occurrence. My background in econometrics has revealed itself to be a potential stumbling block (sometimes; other times it can be an advantage) and I would like to see a list of other potential major differences between econometrics or social science statistics and industrial statistics that I should be aware of. What else is something I shouldn't say is true because it's a concern in econometrics since it's not a concern in industrial statistics?

Answer 1

Firstly, good on you for reflecting on past errors in your consulting work. You obviously care a lot about giving good advice to your clients, and your self-criticism is likely to make you a good practitioner in the long run. People hiring graduate students as statistical consultants ought to be aware that you get what you pay for, and they are hiring someone who is still in their formal education phase.

Statistical theory ought to be thought of as a single body of knowledge on inference, prediction, etc. While it contains a number of different philosophies, methodologies and models, the whole point of the field is that it abstracts away from particular applied problems, and is therefore applicable to any of the sciences when they involve inference, prediction, etc. Statistical theory is (or at least ought to be) the same across applied fields, but there is certainly a difference in emphasis and applications in different applied fields, due to the different kinds of questions and information constraints. Irrespective of the particular fields you are working in, here are some crucial things to bear in mind.

• Always consider/scrutinise the sampling method: Some fields have data that is collected through nice sampling methods that accord with simple statistical model assumptions, so it is easy to get complacent if you practice exclusively in a field like this. For consulting work, it is important to remind yourself that you must consider/scrutinise the sampling method, in case it raises issues (e.g., informative sampling problems). In econometric work, many of the variables under study are macroeconomic estimates from large-scale survey work, where the estimates use a lot of underlying data (e.g., data from a census, tax agency, etc.). In the social sciences much of the research is based on smaller-scale surveys or other small-scale sampling methods, and sometimes these do not involve properly randomised sampling. Make sure you always turn your mind to the sampling method when you meet a new statistical problem.

• Be mindful of the distinction between causal inference versus predictive inference: In predictive inference we only care about the predictive accuracy of a model/method, and we do not care much if an intermediary statistical association is estimated poorly or whether a statistical association is due to a causal relationship or not. In general, these problems are "easy". Contrarily, in causal inference we care about the causal effect of variables and so it then becomes important to consider whether specific parameters in a model are estimated well, and whether any statistical association detected is due to a causal relation or not. Generally this involves having knowledge of experimental theory, particularly the distinction between controlled and uncontrolled experiments. Causal inferences can be made from randomised controlled experiments, and in uncontrolled cases we usually fall back on trying to "control" for unobserved confounding factors using regression methods. Learn as much as you can about experimental methodology, and also learn to judge how well (or badly) uncontrolled experiments can make causal inferences through controlling for confounding variables.

• Always consider problems of "over-fitting": Your general tendency to include model terms even if they don't pass individual "inclusion tests" (and your general skepticism of stepwise regression) is quite reasonable in my view, and accords well with the wisdom of the profession. Methods like that often lead to "over-fitting", so it is often reasonable to throw in a bunch of model terms and keep them in the model even if some look like they might be irrelevant.

• Keep an eye on the "big picture": Sometimes in consulting work (particularly in the social sciences) statisticians can agonise over squeezing every last drop of information out of a small sample, and questions of whether to include or exclude a single model term then seem like a big deal. If you find this is happening, it may be a sign that the sample size is too small to give reliable and robust inferences (without a heavy dependence on model choices) and the best advice may be that your client should get more data. Sometimes statisticians are loath to propose more data as the answer to a problem, since the essence of our subject is to make the best possible inferences/predictions from what information we have, but in some cases the sample size is the "big picture" and the inclusion/exclusion of a model term is the "small picture".

Answer 2

I assume by "industrial" statistics you mean 'applied' statistics, where the theory behind it is not as relevant. But as the above reply says, all fields of statistics SHOULD be more-or-less the same, although some emphasize different things than others.

My own work is kind of in-between academic, social-science-type of statistics and applied industrial statistics. I do a lot of data exploration (biomedical data) and often end up doing regressions and things in an attempt to both explain and predict outcomes. My advice is do indeed include variables in your regression models that may not be statistically significant IF they improve model fit (or if taking them out reduces model fit). Two little diagnostics I make use of (talking about linear regression here) is the R-squared value and the standard error of the regression. The R-squared shows how much the predictors explain the outcome variable, i.e., how well one accounts for the other. If you keep adding variables into your model, the R-squared ALWAYS goes up, even if those variables have no real value and don't predict things any better. The standard error of the regression is the average deviation from the regression line that your outcome variables have; i.e., how "tight" the points are around the line. Both are often overlooked in regression, and both are very useful.

Some other tips of the trade:

1. Sample size matters a lot; often a result won't be significant NOT because of the effect size, but because of the sample size. Become familiar with statistical power (to detect a significant result) and use it in your reports as necessary to larger sample sizes in the future and to qualify your results.
2. Graph your results. And intermediate steps. A picture is worth a thousand words. And that includes getting skilled with graphing in your software.
3. Learn new techniques often. There are free and paid forums and statistical learning sites on-line. Use them. As Julia Child said, "learn every time you cook." And in statistics, you are cooking numbers! Scan the literature and see how others are presenting their results; you'll get good ideas for data presentation at least.
4. Keep improving your writing. The write-up of your results is vital. People hear what they want to hear sometimes, so learn to phrase things carefully, especially any caveats or assumptions or violations of assumptions about the statistical models you use (which isn't always a deal-breaker).
   4a) Publish something once in a while, even if it's just a minor technical finding or example in your field with a colleague as co-author. It boosts your credibility and is helpful to get your next job. And keep your resume updated at all times.
5. Keep it simple. The simplest solutions (for example in regression) are often the best for practical utility.
6. Learn bootstrapping. I've just become familiar with it and it is intriguing. It may be quite useful for small samples.
7. Learn TWO statistical software packages. I don't know what field you're in exactly, but I recommend R and SAS as one of them. (I'm an SPSS man myself, but I could probably earn literally twice as much money doing the same kind of work if I had learned SAS instead. They don't teach you those lessons in graduate school).