(Linear regression) Can I train and validate at the same time using the following approach?

Question

In a lot of material I found online, training and validation seems to be an iterative process

For example, the regularized regression problem

$E = \|Xw - t\|_2^2 + \lambda \|w\|^2_2$

$X$ is data matrix, $w$ is weights of linear predictors, $t$ is targets.

Their algorithm seems to be,

• First, we find $w^\star$ by minimizing $E$ for some $\lambda$,

• Next, use $w^\star$ to find the error on validation set

• Then solve $w^\star$ again by minimizing $E$ for some different $\lambda$,

• Next, use $w^\star$ to find the error on validation set

  $\vdots$

• Pick the best performing $w^\star$ on the validation set.

What I have in the my code is to compute several weights $w^\star_k$ at the same time, and pick the best one based on validation set.

I want to perform training and validation in one shot

• Find $w^\star_1, \ldots, w_N^\star$ ($N$ different predictors) by minimizing $E$ on training set for some $\lambda_1, \ldots, \lambda_N$ ($N$ different regularization constants),
• Run all of $w^\star_1, \ldots, w_N^\star$ on validation set.
• Pick best $w^\star_k$, $k \in \{1, \ldots, N\}$

Is this a proper way of choosing my hyperparameter?

Sorry if this seems to be a basic question. First time doing "validation"/hyperparam tuning.

Answer 1

All the calculations are the same, wether you are doing it sequentially or in one shot, so yes. Remember, if you want to asses the out of sample error you should keep out an untouched test set and evaluate the error on that. The error that you get on the validation set with the picked w will underestimate the true out of sample error, as this is was chosen specifically to minimize the error on the validation set

Answer 2

Do you use python? do the hyperparameter tuning by RandmizedSearchCV or GridSearchCV: https://scikit-learn.org/stable/modules/grid_search.html

As you're doing Ridge regression https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Ridge.html, Should be very straight forward.

RandmizedSearchCV or GridSearchCV learn different models given deterministic hyperparameters or a random approach of the hyper parameters. So you can pick your favorite lambdas and give it a go.

The output of the hyperparametersearch is (if I remember correctly) the loss calculated by cross validation (therefore CV at the end) for each model and the models itself.

Although this technique is learning one-by-one, for you it should feel parallel as you only run it once.