# Determining which combination of independent variables has the most impact on a dependent variable

I have a black box function $f:D\subseteq\mathbb{R}^{n}\to [0, 1]$ that I wish to model (or at least find a point in $x\in D$ so that $f(x)$ is as close to $1$ as possible). I have begun by randomly sampling $D$ so that I have $x_1, ..., x_m\in D$ with $f(x_1)=y_1, ..., f(x_m)=y_m$.

Question: Is there a way that I can find a vector (or vectors) in $\mathbb{R}^{n}$ that point in a direction (or directions) that maximize the variability of the outputs? Basically, what I would like is a vector (or set of vectors) that work like a gradient; but, instead of pointing in the direction that the function increases the fastest, I want the vectors to point in a direction that causes the output to vary the most.

I am (somewhat) familiar with PCA and this question is inspired by it. Essentially, what I'd like to do is find a plane (maybe affine plane) on which most of the variability of $f(x)$ lies. The motivation is that, on that plane, I must sample more heavily if I want to understand what $f(x)$ really looks like. Here I'm assuming that $f(x)$ is relatively boring (constant) off the plane (or hyperplane), but on that plane it is interesting (much more variable). PCA almost accomplishes this. If I were to imagine my inputs $x_1, ..., x_m\in \mathbb{R}^{n}$ I could use PCA to, in a sense, describe a lower dimensional space in which my inputs lie (with vectors pointing in the directions that maximize the variation of my inputs). BUT, I want a direction (in $\mathbb{R}^{n}$) which maximizes the variation in my outputs.

Searching this site, I found this question which has a nice answer but is somewhat incomplete (for my goal at least). The solutions here appear to either (a) determine how much a single variable effects the output--this is equivalent to only allowing direction vectors of the form $\langle 0, 0, 1, 0, 0, 0\rangle$ whereas I want to allow more general vectors-- or (b) to apply a modeling technique such as linear regression or the lasso.

There are also other questions which appear to be similar to mine here and here. I suspect that a solution to my question may also answer these questions. Unfortunately, those questions are somewhat old and unanswered.

• Have you tried a generic root-finding or optimization method, such as R's optim function? – Kodiologist Oct 14 '16 at 21:36
• @Kodiologist That's not really feasible as it takes anywhere from 10-30 minutes to query the black box function and it isn't clear whether or not the function is differentiable or even continuous. – TravisJ Oct 14 '16 at 22:26