0
$\begingroup$

Let's say for a text mining problem (e.g creating a predictive model using text analysis), using a feature selection method (e.g TF-IDF) we come up with 1000 features/words/tokens.

Is there some principals that suggest what number of features we should use? What I currently do is creating the model based on all 1000 features, then reducing the number of features until reaching the highest accuracy when doing train and test.

Is it the correct way to deal with number of features or is it somehow fishing a model which might result in a non-robust model?

$\endgroup$
0
$\begingroup$

It is usual to use all sorts of feature derivation you think are probably useful for your problem, then apply feature selection, feature reduction, and similar concepts to make best use of those features in your models. Not using a helpful feature derivation in the first place causes this information to be missing for your model, which will likely cause worse-than-possible results. So your feature selection and reduction approaches should account for not-helpful and redundant information (this is frequently coupled with training/evaluating models to avoid overfitting/selection bias).

Concerning robustness of the resulting model: this should rely on using a thorough evaluation approach and choosing a reasonable simple model from all evaluated models - instead of limiting feature derivation in the first place. Using repeated CV (training and evaluating different model types and parametrizations, then selecting best model based on resample performances) together with data partitioning appropriate for your setup (completely held-back test partition for estimating performance of final model, poss. done population independent like leave-subject-out-CV) will be OK for most setups.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.