What does Google mean with Rule 21 in their machine learning course? Nº of vars you give your model access to or Nº in your fully trained model?

Question

When limiting the number of features in model training due to the size of your data, do I need to limit the number of features in the training data, or just the number of features selected in the final model (e.g as can be done through regularisation)?

I'm asking the question because of some unclear guidance provided by google in their machine learning crash course: https://developers.google.com/machine-learning/guides/rules-of-ml/#rule_21_the_number_of_feature_weights_you_can_learn_in_a_linear_model_is_roughly_proportional_to_the_amount_of_data_you_have

It states:

Rule #21: The number of feature weights you can learn in a linear model is roughly proportional to the amount of data you have. The key is to scale your learning to the size of your data:

• If you are working on a search ranking system, and there are millions of different words in the documents and the query and you have 1000 labeled examples, then you should use a dot product between document and query features, TF-IDF, and a half-dozen other highly human-engineered features. 1000 examples, a dozen features.

• If you have a million examples, then intersect the document and query feature columns, using regularization and possibly feature selection. This will give you millions of features, but with regularization you will have fewer. Ten million examples, maybe a hundred thousand features.

Thank you

Answer 1

The key sentence is

The number of feature weights you can learn in a linear model is roughly proportional to the amount of data you have.

Which tells us that the thing the authors are writing about is the number of parameters under estimation (in a linear model). If you follow this advice, you limit the number of features in your training data.

I agree that this isn't a particularly precise or well-written blog post. This passage is a bit unclear.

If you have a million examples, then intersect the document and query feature columns, using regularization and possibly feature selection. This will give you millions of features, but with regularization you will have fewer. Ten million examples, maybe a hundred thousand features.

I think the intended meaning is this: if we take as given that we want a smaller number of features than the number of samples, and we know the ratio should be roughly several orders of magnitude, then we must know that "using regularization and possibly feature selection" is applied to the whole of the feature space because millions of features with to millions of observations implies roughly a 1:1 ratio. After this regularization, you'll have fewer ("with regularization you will have fewer").

Apparently, the authors do not reckon that the regularization or feature selection procedure involve estimating feature weights in a linear model.