I have a fairly large space of feature variables in which I want to build a predictor for a target variable. My input dataset for training the predictor are sampled from the space using a mix of log and uniform priors (related to the expected physics of the underlying problem). The distribution of sampled points is not a priori representative of the structure of the target variable.
I could throw all of this directly into some machine-learning tool, but to make it tractable I first want to reduce the dimensionality of the feature space to those variables on which the target has a strong dependence (NB. not strong correlation per se, since dependences may not be monotonic; maybe more like large mutual information). The equivalent is commonly done by identifying eigenvectors with PCA, but these reflect the "physical" variance of samples in the feature space -- the thing I put into my sampler by hand -- rather than the dependence on the target.
The question: Is there a method for identifying principle directions that maximise target sensitivity, rather than the feature-structure of the sample dataset? I feel like I must be missing something obvious!