0
$\begingroup$

I have a fairly large space of feature variables in which I want to build a predictor for a target variable. My input dataset for training the predictor are sampled from the space using a mix of log and uniform priors (related to the expected physics of the underlying problem). The distribution of sampled points is not a priori representative of the structure of the target variable.

I could throw all of this directly into some machine-learning tool, but to make it tractable I first want to reduce the dimensionality of the feature space to those variables on which the target has a strong dependence (NB. not strong correlation per se, since dependences may not be monotonic; maybe more like large mutual information). The equivalent is commonly done by identifying eigenvectors with PCA, but these reflect the "physical" variance of samples in the feature space -- the thing I put into my sampler by hand -- rather than the dependence on the target.

The question: Is there a method for identifying principle directions that maximise target sensitivity, rather than the feature-structure of the sample dataset? I feel like I must be missing something obvious!

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ If you're just looking for features that are relevant to the outcome, you can use methods like Boruta or any other feature selection tool. If you absolutely need decorrelated inputs, then applying PCA after Boruta would seem to fit the bill. But I don't know what "PCA-like directions" means when you're expressly interested in (1) nonlinearity and (2) sensitivity to a target variable because PCA does neither of those things. $\endgroup$
    – Sycorax ♦
    Commented Aug 16, 2019 at 14:56
  • $\begingroup$ @Sycorax Thanks -- just the name Boruta is already a useful piece of info, and it seems there's a Python implementation. Although if I understand correctly from your post and a quick Google, it doesn't do any diagonalisation/decorrelation; I don't see how PCA after Boruta would help, since it'd have the same problems I mentioned in the OP. So it doesn't immediately seem much of an evolution from just identifying features that are highly correlated with the target (modulo issues with correlation). I can think of an inefficient way to do both at once, but was hoping for a standard way... $\endgroup$
    – andybuckley
    Commented Aug 16, 2019 at 16:13
  • $\begingroup$ @Sycorax By "PCA-like directions", I just meant orthogonal linear combinations -- I felt people on this site were likely to feel that they immediately knew the sort of basis I had in mind if I mentioned PCA up-front: maybe not! And I'm not expressly interested in non-linearity in the basis: indeed I want linear combinations that sequentially maximise a metric... just not the metric of projected sample variance. $\endgroup$
    – andybuckley
    Commented Aug 16, 2019 at 16:16
  • $\begingroup$ You're correct that Boruta doesn't give you an orthogonal basis; but it does give you features which are relevant to the outcome, which seems to address the first half of the question. Perhaps the second half is answered by partial least squares? $\endgroup$
    – Sycorax ♦
    Commented Aug 16, 2019 at 16:21

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.