10
$\begingroup$

My objective is to classify sensor signals. The concept of my solution so far is : i) Engineering features from raw signal ii) Selecting relevant features with ReliefF and a clustering approach iii) Apply N.N, Random Forest and SVM

However I am trapped in a dilemma. In ii) and iii), there are hyperparameters like k-Nearest Neigbours for ReliefF or the window length, for which the sensor signal is evaluated, or the number of hidden units in each layer of N.N.

There are 3 Problems I see here : 1) Tuning feature selection parameters will influence the classifier performance 2) Optimizing hyperparameters of classifier will influence the choice of features. 3) Evaluating each possible combination of configuration is intractable.

So my questions are : a) Can I make a simplifying assumption, s.t. tuning feature selection parameters can be decoupled from tuning classifier parameters ? b) Are there any other possible solutions ?

$\endgroup$
  • $\begingroup$ I think decoupling feature selection tuning and classifier tuning is valid, since the heuritsic for reliefF aims to maximize inter-class variance and minimize intra-class variance which also indicates a good classifier. Therefor tuning optimal parameters for reliefF also makes a good classifer more 'likely'. However having a mathematical formulation to back this idea up, would be very nice. $\endgroup$ – Grunwalski Mar 1 '17 at 9:08
  • $\begingroup$ A specific variant of this question: Should feature selection be part of the crossvalidation routine (as in: #for each classifer hyperparam set: #for each k-fold CV run: 1) feature selection, 2) feature scaling, 3) classifier fit 4) predict on test set ? $\endgroup$ – Nikolas Rieble Jan 19 '18 at 8:59
  • 1
    $\begingroup$ @NikolasRieble I just wrote an answer to the original question, and also included your question in the answer $\endgroup$ – Dennis Soemers Jan 19 '18 at 9:48
9
+25
$\begingroup$

Like you already observed yourself, your choice of features (feature selection) may have an impact on which hyperparameters for your algorithm are optimal, and which hyperparameters you select for your algorithm may have an impact on which choice of features would be optimal.

So, yes, if you really really care about squeezing every single percent of performance out of your model, and you can afford the required amount of computation, the best solution is probably to do feature selection and hyperparamter tuning "at the same time". That's probably not easy (depending on how you do feature selection) though. The way I imagine it working would be like having different sets of features as candidates, and treating the selection of one set of features out of all those candidate sets as an additional hyperparameter.

In practice that may not really be feasible though. In general, if you cannot afford to evaluate all the possible combinations, I'd recommend:

  1. Very loosely optimize hyperparameters, just to make sure you don't assign extremely bad values to some hyperparameters. This can often just be done by hand if you have a good intuitive understanding of your hyperparameters, or done with a very brief hyperparameter optimization procedure using just a bunch of features that you know to be decently good otherwise.

  2. Feature selection, with hyperparameters that are maybe not 100% optimized but at least not extremely terrible either. If you have at least a somewhat decently configured machine learning algorithm already, having good features will be significantly more important for your performance than micro-optimizing hyperparameters. Extreme examples: If you have no features, you can't predict anything. If you have a cheating feature that contains the class label, you can perfectly classify everything.

  3. Optimize hyperparameters with the features selected in the step above. This should be a good feature set now, where it actually may be worth optimizing hyperparams a bit.


To address the additional question that Nikolas posted in the comments, concering how all these things (feature selection, hyperparameter optimization) interact with k-fold cross validation: I'd say it depends.

Whenever you use data in one of the folds for anything at all, and then evaluate performance on that same fold, you get a biased estimate of your performance (you'll overestimate performance). So, if you use data in all the folds for the feature selection step, and then evaluate performance on each of those folds, you'll get biased estimates of performance for each of them (which is not good). Similarly, if you have data-driven hyperparameter optimization and use data from certain folds (or all folds), and then evaluate on those same folds, you'll again get biased estimates of performance. Possible solutions are:

  1. Repeat the complete pipeline within every fold separately (e.g. within each fold, do feature selection + hyperparameter optimization and training model). Doing this means that k-fold cross validation gives you unbiased estimates of the performance of this complete pipeline.

  2. Split your initial dataset into a ''preprocessing dataset'' and a ''train/test dataset''. You can do your feature selection + hyperparameter optimization on the ''preprocessing dataset''. Then, you fix your selected features and hyperparameters, and do k-fold cross validation on the ''train/test dataset''. Doing this means that k-fold cross validation gives you unbiased estimates of the performance of your ML algorithm given the fixed feature-set and hyperparameter values.

Note how the two solutions result in slightly different estimates of performance. Which one is more interesting depends on your use-case, depends on how you plan to deploy your machine learning solutions in practice. If you're, for example, a company that intends to have the complete pipeline of feature selection + hyperparameter optimization + training running automatically every day/week/month/year/whatever, you'll also be interested in the performance of that complete pipeline, and you'll want the first solution.

If, on the other hand, you can only afford to do the feature selection + hyperparameter optimization a single time in your life, and afterwards only somewhat regularly re-train your algorithm (with feature-set and hyperparam values fixed), then the performance of only that step will be what you're interested in, and you should go for the second solution

$\endgroup$
  • $\begingroup$ Can you provide references as well? $\endgroup$ – Nikolas Rieble Jan 23 '18 at 9:34
  • $\begingroup$ There are some pictures of a well-known book in this post: nodalpoint.com/not-perform-feature-selection . Those seem to agree with my ''possible solution 1''. I don't have a reference necessarily for the other case, other than... myself? I did provide my reasoning/motivation there, which in my opinion checks out, so that's the reference :D $\endgroup$ – Dennis Soemers Jan 23 '18 at 9:50
  • 1
    $\begingroup$ That chapter of ESL should be 100% required reading for any predictive modeler. $\endgroup$ – Matthew Drury Jan 25 '18 at 20:18
  • $\begingroup$ So regarding soln 1, how do you get your final feature set and model hyperparameters after running feature selection (fs) and hyperparam optimization (ho) in several iters of cv? As well, when we perform these in an iter of cv, do we run fs first, and then ho using those features? $\endgroup$ – skim Jun 14 '18 at 5:45
  • 1
    $\begingroup$ @skim CV is generally used just to get a good estimate of performance. You typically wouldn't directly start using any of the models trained in one of the sets of $K - 1$ folds. If you find the performance as estimated through CV to be satisfactory, you'd run the complete pipeline once more on the full training dataset (including, again, feature selection and hyperparam tuning). The feature set + hyperparams + model you get from that is what you'd put "in production" $\endgroup$ – Dennis Soemers Jun 14 '18 at 8:06
3
$\begingroup$

No one mentioned approaches that make hyper-parameter tuning and feature selection the same so I will talk about it. For this case you should engineer all the features you want at the beginning and include them all.

Research now in the statistics community have tried to make feature selection a tuning criterion. Basically you penalize a model in such a way that it is incentivized to choose only a few features that help it make the best prediction. But you add a tuning parameter to determine how big of a penalty you should incur.

In other words you allow the model to pick the features for you and you more or less have control of the number of features. This actually reduces computation because you no longer have to decide which features but just how many features and the model does the rest.

So then when you do cross-validation on the parameter then you are effectively doing cross-validation on feature selection as well.

Already there are many ML models that incorporate this feature selection in some way or another.

  • Doubly-regularized support vector machines which is like normal SVM but with feature selection
  • Elastic net which deals with linear regression
  • Drop-out regularization in neural networks (don't have reference for this one)
  • Random forest normally does random subsets of the features so kind of handles feature selection for you

In short, people have tried to incorporate parameter tuning and feature selection at the same time in order reduce complexity and be able to do cross-validation

$\endgroup$
0
$\begingroup$

I think you are overthinking quite a bit there. Feature selection, which is part of feature engineering, is usually helpful but some redundant features are not much harmful in early stage of a machine learning system. So best practice is that you generate all meaningful features first, then use them to select algorithms and tune models, after tuning the model you can trim the feature set or decide to use new features.

The machine learning procedure is actually an iterating process, in which you do feature engineering, then try with some algorithms, then tune the models and go back until you are satisfied with the result.

$\endgroup$
  • $\begingroup$ You mean it is trying untill it works :D $\endgroup$ – Grunwalski Apr 20 '17 at 12:57
  • $\begingroup$ Trying in an ML procedure, not randomly. Actually ML is actually a bit of hacking per se. $\endgroup$ – THN Apr 20 '17 at 15:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.