Are the vectorization settings considered hyperparameters in ML?

Question

Short definition of HP:

"In machine learning, a hyperparameter is a parameter whose value is set before the learning process begins. Hyperparameter optimization or tuning is the problem of choosing a set of optimal hyperparameters for a learning algorithm."

Examples of HP: "alpha" in naive bayes, C in SVM, nr. of layers in NN.

I know that one tunes these hyperparameters on a validation set (distinct from the training and test set)

For text classification, however, one vectorizes the text before training; with vectorization there are also many settings (e.g. the max_features function in sklearn's Vectorizer). I found that these vectorization settings greatly affect training,validation & test set performance.

My question is: are the vectorization settings considered hyperparameters, if so, is the fact that we have to define them before training, is that considered a limitation for modelling generalization?

Answer 1

You can view vectorization parameters in exactly the same way as 'normal' hyperparameters. These are parameters you have to fix beforehand and find good ones by evaluating them on validation sets.

The length of your text embedding for example regularizes the expressability of your model. Larger embeddings will give a model more freedom to fit the data compared to smaller embeddings. Similar to regularization (ridge or lasso) in linear regression.