0
$\begingroup$

I use the knn Classifier for a binary classification problem. To improve the classification results I would like to multiply features by weights that are learned from data.

I found different ways to find feature importance values, e.g., the feature_importance of RandomForest or the mutual information value for each feature.

Is it meaningfull to use these values as weights for the features and use them for the distance calculation in the knn classifier?

$\endgroup$
0
$\begingroup$

You can't apply any information used from learning/training to feature values during class prediction. This is called "information leakage" for which some learned information from training is applied to the features to help classification results. In your case, importance scores from RF are based on classification results of RF. Importance scores are used to judge which features are predictive of class, and are not intended to be used as weights for multiplying with feature values.

If you want to improve classification results, try transforming your feature values with mean-zero standardization, normalization, or fuzzifying, or denoising/decorrelating with PCA (dimensional reduction). If those don't help, there is e.g. "boosted decision trees," in which a weak classifier is boosted to improve prediction. There are many boosted versions of a lot of classifiers.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ To clarify: The importance of features (rf) is not learned on the test set but on a split of the training set, so that no test data is used before evaluation of the model on the test set. $\endgroup$ – methus Dec 30 '19 at 10:37
  • $\begingroup$ I wanted to multiply the feature with weights, so that the distance between not important feature will not have a big influence on the whole distance. $d(x,y)=\sum_i (w_i * |x_i-y_i|)$ $\endgroup$ – methus Dec 30 '19 at 10:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.