I am about to embark on a lengthy study for a work project. At the core of it is the need for a statistical tool to help quantify "sensitivity." I am the lead software developer on the task, but I have a pretty strong background in statistics (at least compared with many other software developers). However, my stats background never touched on this idea of sensitivity, especially not outside of an OLS approach.
I have been reading the Wikipedia article on Sensitivity Analysis today, and believe the section on regression analysis touches closely to the problem I am having. Let me explain:
The approach that management wants to take is more or less precisely the "garbage-can" approach from Achen's paper "Let's put garbage-can regressions and garbage-can probits where they belong.". The goal is to make a nice software front-end that gives our researchers a simple interface to select data elements from a bunch of database tables, apply time series lags and/or simple pre-treatments and data cleaning, and then "dump" them in to a regression, with all of the coefficients and summary stats similarly "dumped" out to a big table o' coefficients at the end.
I have been diligent in making this tool and I'm pretty proud of how easy it is to use. Our parameter scripts require very little to be written beyond a simple string such as "DepVar ~ alpha + beta_1*Var1 + beta_2*Var2 + ..." and some simple definitions about what "DepVar" or "Var1", etc., means in the string.
But I also can't help feeling pretty bad that "productionizing" this kind of thing is super misleading. The system is entirely built on ordinary least squares, with no ability to tweak assumptions about heteroskedasticity, error distributions, etc. You cannot swap in other regression techniques for OLS. It's all OLS, all the time, and this was a conscious choice by management.
The output of the system will be a very large set of time-series coefficients on an individual-by-individual basis. So, in down-stream parts of our analysis pipeline, other programs can load a big time series of, say, "Individual X" past sensitivities to some variable. These are used for risk modeling and outlier treatments among other things.
I can't help but feel like this is a very bad way to quantify the sensitivities. Any time there are violations of the OLS assumptions, and especially when these relationships are non-linear, the coefficients can be extremely misleading, even if they have good-looking Frequentist summary stats.
I've raised this issue with my team many times, but the managers in charge seem extremely hell-bent on having simple-to-manipulate statistics. They also seem overly concerned about "interpretability" -- in the sense that they believe these coefficients from OLS can be directly read off as normalized loadings of one variable on another variable. I agree that can sometimes be true, but only if you're really sure that the linear model isn't badly misspecified. And having looked at the data myself, I do believe the linear model is often badly misspecified. I have quantified that and repeatedly shown this well-documented fact, along with the Achen paper, to my managers, but they just will not budge.
Lately I have been wondering: how do other approaches tackle this problem? In a machine learning context, like fitting a SVM model or a decision tree, how do practitioners quantify the sensitivity between the different variables? How does this work in a setting where there may be non-linearities?