I have a dataset of 7 trips on a vehicle, where a component fault occurred on the 4th trip and was fixed following that. The goal is to predict when the fault will occur for similar datasets. The dataset contains 9 parameters of numerical sensor data for each trip.

I was requested to generate statistical features for each parameter, plot them, and pick the best ones which show an observable "pattern" or some form of differentiation between the pre-fault trips, the trip where the fault occurred, and the trips following the fault repair.

Is this method a valid form of feature selection? Isn't one of the benefits of feature selection to pick out features that might not have observable patterns, but still have predictive power?


Feature selection is a statistical search problem. Several strategies have been developed over the years:

  • Sequential forward search - find the most discriminative single feature, build a classifier from that, find the next feature that performs best (on a training set) in combination with the earlier selected single feature, add the most suited 3'rd feature, and so forth;
  • Sequential backward search - build a classifier from all $n$ features, try removing each feature, and build each possible classifier from $n-1$ features, choose the best performing classifier (on a training set) - and its feature subset, and iterate again by removing the least important feature from that classifier, and so forth;
  • Branch-and-bound search - an algorithmic approach to narrow in on the optimal feature subset, taking some variation into account
  • Floating search - switch between sequential forward search and sequential backward search, according to a specialized algorithmic scheme;
  • Markov Chain Monte Carlo approaches - construct a Markov chain where features are added and removed from the classifier in the current Markov state (Metropolis Hastings algorithm);

The 'sequential' methods are not guaranteed to provide the optimal (sub)set of features because overfitting to noisy features often occurs. This situation is called Peaking. Floating search and MCMC approaches use heuristics and stochastics to perform a directed search for the optimal subset of feature variables.

It seems like you are doing sequential forward search, as appears from your question.

You can start off by the sequential methods and switch to the more advanced approaches if there are clear signs of a performance peak, as estimated from an independent test set.

  • $\begingroup$ How would you determine the most discriminative single feature? Through pure observation of data? $\endgroup$ – cheap brent crude Jul 19 '20 at 13:46
  • $\begingroup$ Also, would a feature, being the most discriminative, that has the best performance on a model also give the best performance on all other models? $\endgroup$ – cheap brent crude Jul 19 '20 at 14:05
  • $\begingroup$ You should build a classifier from one single feature - and test the performance of that. With n features, you this way build n different classifiers. $\endgroup$ – Match Maker EE Jul 20 '20 at 9:30
  • $\begingroup$ A feature that performs best with one type of classifier does not necessarily be the best performing one for a completely different type of classifier. Decision trees from ID3 work very differently than a deep learning neural network or a support vector machine. Hence that feature rankings will often differ for different types of classifiers. $\endgroup$ – Match Maker EE Jul 20 '20 at 9:34
  • $\begingroup$ Sorry if I don't make sense as I'm very new to machine learning. I would like to pick the best set of n classifiers using a correlation algorithm, like so: en.wikipedia.org/wiki/… Then apply PCA to further reduce dimensionality and then a classifier. Would it be better to build n different single feature classifiers instead? $\endgroup$ – cheap brent crude Jul 21 '20 at 8:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.