0
$\begingroup$

As a newbie, I'm not sure if this situation can be described more succinctly using better terminology, so kindly bear with me.

The performance of a machine+human system is dependent on four independent variables, $x_1$ to $x_4$, but the exact relationship is not known beforehand. $x_1$ and $x_2$ are continuous variables, $x_3$ and $x_4$ are discrete.

After observing system performance for a while, I, a stats newbie, ran a multiple linear regression on a training dataset which threw up great correlations (two positive and two negative) with great p-values (e-11 or better) and great adjusted $R^2$ (> 0.85). Test dataset prediction errors were also small (based on normalized RMSE values).

Based on these results, I'd like to optimize system performance by focusing on the variable that has greatest impact, positive or negative, on performance. Because retraining of humans is involved, it is not feasible to focus on more than one variable at a time.

Question: How do I pick the variable that has the greatest impact on performance? Is it simply a matter of picking the one with the greatest absolute value coefficient? Won't it also depend on the the range of each variable? For example if the equation (result of regression) is:

$P = c + 2x_1 - 3x_2 + 4x_3 - 5x_4$

where

$x_1$ varies from 100 to 120

$x_2$ varies from 80 to 105

$x_3$ varies from 40 to 50

$x_4$ varies from 2 to 12

then the maximum absolute impact of $x_4$ is $ 5 * 12 = 60$, $x_3$ is $200$, $x_2$ is $315$ and $x_1$ is $240$.

Should I be using this logic to pick $x_2$ as the variable to focus on?

Update after comments: To optimize performance in this case is to increase the value of P in the sample equation by attempting training (of humans) so that the value of the most impactful variable moves in the direction that improves P. Please ignore cost considerations. Assuming that the potential impact on P is the only consideration, how do I pick one variable to focus on?

$\endgroup$
8
  • $\begingroup$ It would depend on what it might mean to "optimize performance" and "focusing on a variable." Could you explain? $\endgroup$
    – whuber
    Commented Aug 11, 2022 at 19:32
  • $\begingroup$ To optimize performance in this case is to increase the value of P in the sample equation by attempting training so that the value of the most impactful variable increases or decreases, as the case may be (positive or negative correlation). $\endgroup$
    – ottodidakt
    Commented Aug 11, 2022 at 20:05
  • $\begingroup$ Then great consideration must be given to how much each variable can be increased and what it might cost to increase it. If you don't account for those factors, you have no hope of obtaining a good answer. Not to put too fine a point on it -- and to deploy an extreme example in the interest of making this clear -- what if by somebody's mathematical or statistical definition sex or human age were to be the "most impactful" variable? $\endgroup$
    – whuber
    Commented Aug 11, 2022 at 21:11
  • $\begingroup$ Agreed. But assuming that the costs are reasonable and that I only expect to improve the variable within the bounds exhibited in the analysis dataset, my question remains. $\endgroup$
    – ottodidakt
    Commented Aug 11, 2022 at 22:01
  • 1
    $\begingroup$ Without cost considerations, the question is unanswerable because any variable that is not inherently bounded can be changed by an arbitrary amount to induce an arbitrarily large change in the response (under the doubtful but necessary assumption that the model reflects a causal relationship in the first place). $\endgroup$
    – whuber
    Commented Aug 12, 2022 at 12:46

1 Answer 1

4
+50
$\begingroup$

Naively speaking you're looking for what's called feature importance in linear regression. Here's a post about ways to calculate it. One way to do it is to normalize all the features $x_1...x_4$ to the same range of 0-1 like so:

$x_i' = \frac{x_i-min(x_i)}{max(x_i)-min(x_i)} $

Then either rerun the regression or transform the coefficients of your current model. Now your coefficients are ranked by the change in the target for a percentage increase in the feature inside your range.

Then you can just look at the variable with the largest coefficient as "the most important feature" assuming you can increase the value of all the features equally.

Now to the not naive part: what you actually want to do, is maximize the target $p$. But it doesn't depend only on the most important feature, but also on how much you can increase each feature. So what you actually want is to write how much you can increase every feature. for example: I can increase $x_1$ by 2 units, leading to $\Delta x_1 = 2$ and assume: $\Delta x_2 = 1$ $\Delta x_3 = 3$ $\Delta x_4 = 0.0$

if we can't actually increase $x_4$ then obviously there's no point trying. for the other variables in this case we can expect the change in $p$ to be: $\Delta p_1=2*2=4$ $\Delta p_2=-3*1=-3$ $\Delta p_3=4*3=12$

So the most benefit would be from training employees to work on feature 3, then feature 1. If I assume different ability to improve the feature values, like so:

$\Delta x_1 = 1$ $\Delta x_2 = -5$ $\Delta x_3 = 2$ $\Delta x_4 = -2$

we get different results: $\Delta p_1=2*1=2$ $\Delta p_2=-3*-5=15$ $\Delta p_3=4*2=8$ $\Delta p_4=-5*-2=10$

I'd then select to work on feature 2, then feature 4, then 3 and finally on 1.

So inherent to your problem is how much you can change every feature.

$\endgroup$
1
  • $\begingroup$ Great answer. Thank you very much for taking the time to write it. $\endgroup$
    – ottodidakt
    Commented Aug 17, 2022 at 3:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.