Someone told me that the performance of a machine learning model tends to be more sensitive to changes in a small subset of hyperparameters than other. Is this true? Can someone give me an example of this?
Thank you,
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
0
This paper outlines an experiment to discover the most important parameters for random forest, SVM and adaboost. It quantifies how important each parameter is using an experiment, and finds that most of the variation in algorithm performance is contributed by a few parameters in these algorithms.
Jan N. van Rijn and Frank Hutter. "Hyperparameter Importance Across Datasets."
With the advent of automated machine learning, automated hyper-parameter optimization methods are by now routinely used in data mining. However, this progress is not yet matched by equal progress on automatic analyses that yield information beyond performance-optimizing hyper-parameter settings. In this work, we aim to answer the following two questions: Given an algorithm, what are generally its most important hyper-parameters, and what are typically good values for these? We present methodology and a framework to answer these questions based on meta-learning across many datasets. We apply this methodology using the experimental meta-data available on OpenML to determine the most important hyper-parameters of support vector machines, random forests and Adaboost, and to infer priors for all their hyper-parameters. The results, obtained fully automatically, provide a quantitative basis to focus efforts in both manual algorithm design and in automated hyper-parameter optimization. The conducted experiments confirm that the hyper-parameters selected by the proposed method are indeed the most important ones and that the obtained priors also lead to statistically significant improvements in hyper-parameter optimization.
