I have a quick question regarding train, dev, test set. We have a simple log linear model trained via gradient ascent on the log likelihood of the data combined with l2-regularizer. Our instructor said that the dev set should be used to tune the hyperparameters but did not provide any details. I'm not sure how training instances would provide any information as to how to tune the l2 parameter. Can you guide me towards the right direction?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
It's quite simple: you train the model on the train set and evaluate it on the dev set. You repeat this operation for different sets of hyperparameters until you find the most satisfying solution. To do this, you can either use grid search (compare many different combinations of hyperparameters), but there are much better solutions like random search, successive halving, or Bayesian optimization where an algorithm decides on what values of hyperparameters are promising to compare, etc. When you find the best combination of hyperparameters you evaluate the result on the test set because if you tried many combinations of hyperparameters on the dev set, you could end up overfitting to the dev set, having a held-out test set prevents this.
