Should a feature importance score be invariant to transformations of the response? This is more of a philosophical question that came up in a discussion with a friend - consider some 'feature importance' procedure associated to a model (say a regression model). You run your model and then compute feature importance scores for each of your $p$ covariates, and you get $p$ numbers in $[0,1]$ say.
Then, you transform your response $g(Y)$ and repeat the process, computing feature importance scores again.
Question: Is it desirable (of a feature importance procedure) that the two sets of scores are the same ? That is, should a feature importance procedure ideally be invariant to transformations of the response? Or should a feature importance procedure merely give you a score for each covariate relative to all other covariates in the model, and so should just maintain the relative ranks of the feature importances.
 A: I think this is a very hard ask for general $g$.  If you restricted to, say, nonconstant linear $g$, and you want normalized importances, then it seems reasonable.
For a really pathological example, let $y=x_1+1000x_2$, with $x_i\in\{0,1\}$. Clearly $x_2$ is significantly more important.  But let $g=\operatorname{parity}: \mathbb{Z}\to\{0,1\}$, and now $g(y)=g(x_1)$, and $x_2$ is irrelevant to any new model.
For something more continuous, consider instead $y=(2+0.1x_1+0.1x_2)^{x_3}$ with $x_1, x_2$ small scale and $x_3$ discrete in $\{1, 2\}$, and $g(y)=y^3$.  Then $x_3$ "ought" to be the most important for both models, but its importance "ought" to be even greater in the transformed model, I think.
For nonconstant linear $g$, consider the basic case of a linear regression.  Scaling $y$ will just scale the coefficients by the same amount, and a shift will get absorbed by the intercept term.  Normalizing coefficients to get "importances" (multiplying by the standard deviation of the $x$ which hasn't changed, and dividing by the standard deviation of $y$, which has also scaled) washes out the change.
