Short definition of HP:

"In machine learning, a hyperparameter is a parameter whose value is set before the learning process begins. Hyperparameter optimization or tuning is the problem of choosing a set of optimal hyperparameters for a learning algorithm."

Examples of HP: "alpha" in naive bayes, C in SVM, nr. of layers in NN.

I know that one tunes these hyperparameters on a validation set (distinct from the training and test set)

For text classification, however, one vectorizes the text before training; with vectorization there are also many settings (e.g. the max_features function in sklearn's Vectorizer). I found that these vectorization settings greatly affect training,validation & test set performance.

My question is: are the vectorization settings considered hyperparameters, if so, is the fact that we have to define them before training, is that considered a limitation for modelling generalization?


1 Answer 1


You can view vectorization parameters in exactly the same way as 'normal' hyperparameters. These are parameters you have to fix beforehand and find good ones by evaluating them on validation sets.

The length of your text embedding for example regularizes the expressability of your model. Larger embeddings will give a model more freedom to fit the data compared to smaller embeddings. Similar to regularization (ridge or lasso) in linear regression.

  • $\begingroup$ Thanks! however, in the official manuals I see that, e.g, classifiers are fit on the training data, then HP's are tuned on the validation set; however, are the vectorization simply HP's that you have to set before training? Because there is no other way to do it? $\endgroup$ May 25, 2020 at 18:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.