I have constructed a machine learning model (it is similar to Naive Bayes) within the Bayesian framework, and as such, have must select priors.
In my brief exposure to Bayesian statistics, I was told that I can't simply fit my hyperparameters to best "explain" the data, as that will increase my certainty beyond what it should be. I know there is a field, Empirical Bayes, which deals with ways to derive hyperparameters from the data which do not cause the issues that directly fitting it does. However, I think my issue is a little different, as I will not always have data for every single parameter; some parameters' priors may never be updated, and will retain only the information provided by the hyperparameters. The parameters in this case are the weights of the model, which I am planning to all give the same prior.
My question is as such: is it appropriate to use cross validation or a similar technique in order to pick the hyperparameters that best predict the testing data in a ML model? Or should I stick to something minimally informative such as Jeffreys/Reference?
My objective with developing posteriors on my parameters is so that I have a means of quantifying their uncertainty, which I then intend to use as weights.
I will be hanging out on the site for a while if more information is required.