In a model with several parameters, which one should be tuned via cross validation first?

Question

I have a loss function like

$$\eqalign{ L(U, V, P, Q) = & \alpha_1 (R - U \cdot V^T )^2 + \alpha_2 (D - U \cdot P^T )^2 + \alpha_3 (S - V \cdot Q^T )^2 \\ &+ \lambda_1 (\parallel U \parallel^2 + \parallel V \parallel^2) + \lambda_2 \parallel P \parallel^2 + \lambda_3 \parallel Q \parallel^2 }$$

where $\alpha_{1,2,3}$ are weights and $\lambda_{1,2,3}$ are regularization parameters.

Now I'd like to find the best parameters via cross validation, but there are two questions:

Question 1. Should I tune the weight parameter first, or regularization parameter first? Does the order matter?

Question 2. Among weight parameters, which one should I tune first, and which value should I fix for the rest?

Because I thought these parameters are dependent on each other, say $\alpha_1$ and $\alpha_2$, if I fix $\alpha_2$ to some value (and I also don't know which value it should be), and then tune $\alpha_1$, how can I make sure the tuned $\alpha_1$ is globally optimum?

Answer 1

As these hyperparameters interact with each other, it is best to tune them together. Generally, a hyperparameter response surface is very complex, which means that tuning parameters separately usually leads to poor results.

The standard approach to tuning is grid search, e.g. test predetermined tuples of hyperparameters (with cross-validation) and use the one that yielded best performance. Grid search, however, is inefficient. Grid search is very inefficient when you have a lot of hyperparameters (6 is already a problem for grid search). An alternative is random search, which essentially means trying a set of random tuples.

The best option is to use specialized libraries that provide automatic solvers to optimize hyperparameters. These solvers require far less parameter tuples to be tested, and hence require less time. You can find such solvers in Optunity and Hyperopt.