When limiting the number of features in model training due to the size of your data, do I need to limit the number of features in the training data, or just the number of features selected in the final model (e.g as can be done through regularisation)?
I'm asking the question because of some unclear guidance provided by google in their machine learning crash course: https://developers.google.com/machine-learning/guides/rules-of-ml/#rule_21_the_number_of_feature_weights_you_can_learn_in_a_linear_model_is_roughly_proportional_to_the_amount_of_data_you_have
Rule #21: The number of feature weights you can learn in a linear model is roughly proportional to the amount of data you have. The key is to scale your learning to the size of your data:
- If you are working on a search ranking system, and there are millions of different words in the documents and the query and you have 1000 labeled examples, then you should use a dot product between document and query features, TF-IDF, and a half-dozen other highly human-engineered features. 1000 examples, a dozen features.
- If you have a million examples, then intersect the document and query feature columns, using regularization and possibly feature selection. This will give you millions of features, but with regularization you will have fewer. Ten million examples, maybe a hundred thousand features.