As a rookie to machine learning area, I tried to play some Data Science tutorials and beginner competitions to gain some knowledge and experience.

The problem I encountered in every scenarios is that, after choosing an suitable model and searching for optimal hyper parameters. I don't know what I can do to improve the precision or performance.

There are always context specific wisdom behind every tasks to discover useful information. But I wonder if there is a systematically way or a heuristic which can apply on general cases.

I heard of "Feature Engineering" and read some articles about it, but I still don't know how can I apply the knowledge.

The only technique I learned is Recursive Feature Elimination, but usually it only brings little improvement.

When the feature is anonymous or the feature has thousands of dimensions, what's the general strategy to deal with the features?

Is there other directions I can work on?


This is a very general question about enhancing performance of a machine learning model. It is always data dependent to decide on the best approach to improve precision or performance.

As you point out in your question, feature selection (FS) is one of the best solutions to fine tune performance of a learning model. In general, you need to define a search method and a scoring function for an appropriate FS. For example, you can use genetic algorithms to find a combination of features that optimizes precision over training or validation data. Another example is the one you mention i.e. recursive feature elimination RFE where the search method is backward elimination and the scoring function is based on performance of an SVM classifier.

As a general guideline out of a personal experience, I would suggest the following in order:

  1. (Start simple): Measure correlation between your variables or features and if two features are extremely correlated, most probably, you want to keep only one of them.
  2. Figure out if you need to normalize your data and how. Some classifiers for example are more sensitive to how dispersed are the domains of variables from each other.
  3. If you have a very large number of features, you may want to get a clue of how many dimensions are sufficient to explain your data. A good suggestion is to use PCA or other projection methods and if possible, do some visualizations or estimation of performance.
  4. Progress with a more sophisticated feature selection procedure like wrapper based feature selection where you optimize a specific performance metric.
  5. Finally, have a rigorous setup for evaluating performance like cross-validation and be careful from generalizing your conclusions based on only training performance.
  6. Ignore all previous step and resort to deep learning to get an advise on possible new good features (i.e. automated feature engineering). However, this has limitations and was shown to perform well with large amounts of data cases.

These kind of hints are mostly related to variable selection but your data might require sample selection or other pre-processing steps. For example, if you have two classes that are not represented equally (a.k.a. class imbalance problem where one class is explained by many more cases compared to the other one which is explained by only few number of cases), you need to pre-process data and build a training set with equal proportion of informative samples for the classes (note: testing set should be untouched and not pre-processed to allow for an informative practical evaluation of your procedure). Many of the classifiers are affected by this imbalance problem as they intrinsically, optimize accuracy and not precision.

I hope these hints can help but you get a better answer with more details about your problem domain, data specifications and intended tasks or goals.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.