0
$\begingroup$

I've seen several people doing prefiltering based on correlation coefficients. But would it be ok to find best features using GradientBoostingRegressor and then use those to train a network?

Is there any chance I could implement such computation within the Network?

My data consists of 40 thousand features and around 60 samples. All of them numerical and the output is also numerical (the age of the person). It is a regression problem.

I am experiencing "the curse of dimensionality" and wanted to know what would be the best way to address feature selection. Filtering methods and PCA didn't give optimal results.

Improve this question
$\endgroup$
5
  • $\begingroup$ Any pre-processing steps, (including feature selection), that uses the target variable to make the selection risks overfitting. Instead, you should do something like a cross-validation for the whole process. See the discussion about "phased" analysis pipelines here stats.stackexchange.com/a/184023/22311 in the context of L1/L2 regularization. $\endgroup$
    – Sycorax ♦
    Dec 12, 2022 at 3:26
  • $\begingroup$ Hi @Caterina. Please add any additional relevant info in your question body rather than comments. $\endgroup$
    – User1865345
    Dec 12, 2022 at 3:29
  • $\begingroup$ Why do you have to do any feature selection? $\endgroup$
    – Dave
    Dec 12, 2022 at 4:13
  • $\begingroup$ @Dave presumably because training the models naively results in massive overfitting; see OP's previous question stats.stackexchange.com/questions/598587/… $\endgroup$
    – Sycorax ♦
    Dec 12, 2022 at 13:56
  • $\begingroup$ In the end I solved that (the question you mention) by simply changing the activation function of last layer to LeakyRelu. And yes I made a MAE vs number of features graph (ordered by pearson correlation coefficient to age) and the performance does change. $\endgroup$
    – Caterina
    Dec 12, 2022 at 14:19

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.