2
$\begingroup$

I have a very basic machine learning question.

My likelihood function includes a parameter $\alpha$ which I set to a fixed value and do not learn from the model, which makes it a "hyperparameter". And I intend to select the optimal value of this hyperparameter using a cross-validation test. Apparently, the value of my likelihood function depends on the value of $\alpha$. So, should I necessarily use the fixed value of $\alpha$ I used in training while computing the test fold log-likelihood?

First I thought that would be the case, but then I remembered that in penalized regression models, the penalty parameter is treated as the hyperparameter, and while computing the test error, we don't include the penalty term, but just the standard mean squared error. This makes sense but also caused me a confusion because such a computation of log-likelihood without including the value of the hyperparameter is not possible in my case.

Any suggestions are welcome!

Improve this question
$\endgroup$
4
  • $\begingroup$ Yes, that is my question, and I agree with you in that we need to use the same hyperparameters as in training the model while computing test error (or likelihood), but then here is my confusion: In regularized regression (say lasso where the penalty parameter $\lambda$ is a hyperparameter), why don't we include the penalty parameter while computing the test error, but we just compute the MSE using learned weights (fitted parameters) without any penalty term? $\endgroup$
    – user5054
    Commented Aug 5, 2018 at 22:41
  • $\begingroup$ Okay thanks, but this does not answer my question. You said in your previous comment that in testing I should use the same hyperparameters as the ones that have led to the fitted values. Then, although penalty parameter is also a hyperparameter, why do we exclude it from the test error computation? $\endgroup$
    – user5054
    Commented Aug 5, 2018 at 23:23
  • $\begingroup$ Thanks, that's helpful. I am not totally clear about "depends on whether the hyperparameter is necessary to compute your performance metric", though. How do I decide that? For example, if my model's loss is a weighted loss such as $\alpha.loss_1 + (1-\alpha). loss_2$ where $\alpha$ is a hyperparameter and $loss_1$ and $loss_2$ are functions of the fitted parameters, then should I compute the test loss using $\alpha.loss_1 + (1-\alpha).loss_2$ or using $loss_1 + loss_2$? $\endgroup$
    – user5054
    Commented Aug 6, 2018 at 6:24
  • $\begingroup$ That model's loss function is used to optimize/train the model. It is not necessarily used to compare the performance of differently trained models. You can use a different loss function for that. Can't you compare predicted/estimated dependent values with the true dependent values in the data without using $\alpha$? Is your target loss function that you wish to optimize by all this unknown (ie depending on $\alpha$)? (optimizing such things as a penalty parameter is not the final aim, it is just a tool to get to less MSE which is the real aim). $\endgroup$
    – Sextus Empiricus
    Commented Aug 6, 2018 at 6:36

1 Answer 1

Reset to default
1
$\begingroup$

The loss function $f$ that is optimized to obtain the model parameters is not necessarily the loss function $g$ that is used in the cross-validation and used to determine the performance of the fitting procedure in the training and testing data (checking for over-/under-training etc.).

Therefore when you use a hyperparameter in the optimization of $f$ it may not always occur as well in $g$.

After training you may throw away the hyperparameters and only use the fitted coefficients as the model to make predictions with new test data (such as in lasso which is about finding coefficients and $\lambda$ is irrelevant and only fine-tuned to find good coefficients).

I believe that, when your are optimizing hyperparameters, then it would not be great to have hyperparameters included in your loss function $g$ that is used to evaluate the performance (like using a weighted loss function $\alpha L_1 + (1-\alpha) L_2$ as you mentioned in comments). This means that you have no fixed idea about what loss function should be used to define good performance and your procedure will be adjusting your ideas about good performance according to your model, rather than adjusting your model according to your ideas about good performance.

Improve this answer
$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.