0
$\begingroup$

I have constructed a machine learning model (it is similar to Naive Bayes) within the Bayesian framework, and as such, have must select priors.

In my brief exposure to Bayesian statistics, I was told that I can't simply fit my hyperparameters to best "explain" the data, as that will increase my certainty beyond what it should be. I know there is a field, Empirical Bayes, which deals with ways to derive hyperparameters from the data which do not cause the issues that directly fitting it does. However, I think my issue is a little different, as I will not always have data for every single parameter; some parameters' priors may never be updated, and will retain only the information provided by the hyperparameters. The parameters in this case are the weights of the model, which I am planning to all give the same prior.

My question is as such: is it appropriate to use cross validation or a similar technique in order to pick the hyperparameters that best predict the testing data in a ML model? Or should I stick to something minimally informative such as Jeffreys/Reference?

My objective with developing posteriors on my parameters is so that I have a means of quantifying their uncertainty, which I then intend to use as weights.

I will be hanging out on the site for a while if more information is required.

$\endgroup$
2
  • 1
    $\begingroup$ In the machine learning tradition, hyperparameters are tuned in some manner by the analyst to optimize model performance. In the Bayesian tradition, the choice of (hyper)prior is used to represent uncertainty about some aspect of the model. I can't pin down which tradition you subscribe to; could you clarify what you're trying to do? Ex: a model from the machine learning tradition is random forest;. I don't know of anyone who places a distribution over the number of variables to try at each split; they just tune that parameter of the model and then treat it as fixed. $\endgroup$
    – Sycorax
    Jan 12, 2016 at 15:54
  • $\begingroup$ The reason I'm bothering with the Bayesian aspects is because I want to have some manner of quantifying uncertainty on each of my parameters, which I will then use as weights for each parameter. I realize this is something that I should have stated initially, and will edit the body. I hope that helps with clarity $\endgroup$ Jan 12, 2016 at 15:58

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.