I am about to embark on a lengthy study for a work project. At the core of it is the need for a statistical tool to help quantify "sensitivity." I am the lead software developer on the task, but I have a pretty strong background in statistics (at least compared with many other software developers). However, my stats background never touched on this idea of sensitivity, especially not outside of an OLS approach.

I have been reading the Wikipedia article on Sensitivity Analysis today, and believe the section on regression analysis touches closely to the problem I am having. Let me explain:

The approach that management wants to take is more or less precisely the "garbage-can" approach from Achen's paper "Let's put garbage-can regressions and garbage-can probits where they belong.". The goal is to make a nice software front-end that gives our researchers a simple interface to select data elements from a bunch of database tables, apply time series lags and/or simple pre-treatments and data cleaning, and then "dump" them in to a regression, with all of the coefficients and summary stats similarly "dumped" out to a big table o' coefficients at the end.

I have been diligent in making this tool and I'm pretty proud of how easy it is to use. Our parameter scripts require very little to be written beyond a simple string such as "DepVar ~ alpha + beta_1*Var1 + beta_2*Var2 + ..." and some simple definitions about what "DepVar" or "Var1", etc., means in the string.

But I also can't help feeling pretty bad that "productionizing" this kind of thing is super misleading. The system is entirely built on ordinary least squares, with no ability to tweak assumptions about heteroskedasticity, error distributions, etc. You cannot swap in other regression techniques for OLS. It's all OLS, all the time, and this was a conscious choice by management.

The output of the system will be a very large set of time-series coefficients on an individual-by-individual basis. So, in down-stream parts of our analysis pipeline, other programs can load a big time series of, say, "Individual X" past sensitivities to some variable. These are used for risk modeling and outlier treatments among other things.

I can't help but feel like this is a very bad way to quantify the sensitivities. Any time there are violations of the OLS assumptions, and especially when these relationships are non-linear, the coefficients can be extremely misleading, even if they have good-looking Frequentist summary stats.

I've raised this issue with my team many times, but the managers in charge seem extremely hell-bent on having simple-to-manipulate statistics. They also seem overly concerned about "interpretability" -- in the sense that they believe these coefficients from OLS can be directly read off as normalized loadings of one variable on another variable. I agree that can sometimes be true, but only if you're really sure that the linear model isn't badly misspecified. And having looked at the data myself, I do believe the linear model is often badly misspecified. I have quantified that and repeatedly shown this well-documented fact, along with the Achen paper, to my managers, but they just will not budge.

Lately I have been wondering: how do other approaches tackle this problem? In a machine learning context, like fitting a SVM model or a decision tree, how do practitioners quantify the sensitivity between the different variables? How does this work in a setting where there may be non-linearities?

  • $\begingroup$ This is in its way a good question, but the zero answers indicates it is to broad. To get answers, you should really try to tell us some more about your applied context! If the question is how to explain management why their ways are wrong, I guess you need to show them a detailed case study where the present methods leads to the wrong conclusion. $\endgroup$ – kjetil b halvorsen Jul 8 '14 at 20:59
  • $\begingroup$ I'm asking here to get feedback on the general problem. Such as, in machine learning sensitivity is measured via such-and-such, but in a classical Frequentist setting it is measured with blah-blah and in a Bayesian setting it's measured via blah... hopefully with references. Also, this is issue is quite old now and I have moved on from the position where the original project happened. I'm still very interested in this issue though ... just not connected to a specific problem instance anymore. $\endgroup$ – ely Jul 9 '14 at 18:31
  • $\begingroup$ Last thought: I did show them the Achen paper that I linked above, which provides pretty comprehensive examples of how the boiler-plate regression approach can go wrong. They simultaneously acknowledged that it could go wrong and that there was not much theoretical reason why it should even work correctly in the first place ... and at the same time essentially said they did not care because they wanted something expressed as regression coefficients regardless of the ramifications of that sort of model. And these were highly educated veterans running a long-standing, successful company. :/ $\endgroup$ – ely Jul 9 '14 at 18:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.