Questions tagged [bayesian-optimization]
Bayesian optimization is a family of global optimization methods which use information about previously-computed values of the function to make inference about which function values are plausibly optima. Its applications include computer experiments and hyper-parameter optimization in some machine learning models.
181
questions
5
votes
1
answer
58
views
Deriving Log Marginal Likelihood for Gaussian Process
I am trying to evaluate the following integral marginalized across all possible functions.
$$\mathbb{P}(y|X,\theta) = \int \mathbb{P}(y|f)\ \mathbb{P}(f|X,\theta) \ df$$
In G.P. we assume prior to be ...
2
votes
1
answer
63
views
Bayesian optimization on a polynomial regression as the surrogate model
My understanding of Bayesian optimization is that it is generally used in conjunction with Gaussian process (GP) as the surrogate model. Because GP inherently produces an uncertainty of estimate, the ...
- 103
0
votes
0
answers
13
views
Is there an alternatives to "one-tailed test" and "two-tailed test" in Bayesian A/B testing approach?
I am trying to grasp Bayesian A/B testing (so far I can't see it looks much different from standard Frequentist approach when using an uninformative prior).
A solution that we use for A/B testing ...
- 131
0
votes
0
answers
21
views
What's a good kernel if I want to use Gaussian Process on grid maze navigation problems?
I am trying to apply gaussian process regression to the problem of grid maze navigation following the approach described in this paper:
Reinforcement learning with Gaussian processes
The grid maze ...
1
vote
0
answers
53
views
Monte Carlo Dropout as surrogate model for Bayesian Optimization
I am interested in using Monte Carlo Dropout as a surrogate model for Bayesian optimization. I noticed that the paper states:
The use of dropout (and its variants) in NNs can be interpreted as a ...
- 11
2
votes
1
answer
114
views
Role of standard deviation in Bayesian optimization using GP
I am new to GP and BO and I have been playing with the two in a simple 1D context which happens to be practically relevant to what I am working on. Essentially, I am trying to find a peak (modeled as ...
- 43
1
vote
1
answer
29
views
Bayesian terminology for experiments
I am learning about bayesian experimental design and confused about the "Bayesian" terminology. Multi-armed bandits are normally in the syllabus of bayesian experiments. But there are ...
- 73
4
votes
0
answers
46
views
Choice of model surrogate for bayesian optimization
I am running BayesSearchCV to optimize the hyperparameters of my machine learning model. This particular procedure allows the user to choose the surrogate model. The options are Gaussian Process, ...
- 924
0
votes
0
answers
12
views
How to go about an optimization problem where posterior denstiy is involved
I am currently reading a paper by Yao et al.(2018) where they discussed about stacking bayesian prediction distribution for K models. I got the idea but I am not sure how you will implement it in ...
- 21
0
votes
0
answers
78
views
Why is optuna not considering previous trial results when sampling new hyperparameter values?
I'm trying to make optuna aware of previously-obtained trial results, following the approach described in the docs. I.e. I create frozen trials based on existing results, add them to the optuna study, ...
- 1
0
votes
1
answer
28
views
Bayesian optimization with constraints
I want to perform Bayesian optimization for a certain physical task but with additional requirements. We have access to a set of variables and want to maximize (multiple) signal outputs from an ...
- 23
1
vote
0
answers
50
views
Why was Bayes better on exponential model rather than log-linear model?
I wanted to train a Bayesian version of this model which we can consider to be this log-linear form.
$$\ln \text{PSI} = \alpha \text{Time} + \beta.$$
Here are the priors I guessed for $\alpha$ and $\...
- 6,115
1
vote
0
answers
25
views
How to apply Bayesian Optimization to a function depending only on categorical variables?
I would like to apply Bayesian Optimization (BO) to a black-box function depending only on multiple categorical variables. In my application, each categorical variable has 3 possible categories. I ...
- 71
0
votes
0
answers
53
views
Expected Improvement (EI) in Bayesian Optimization failed to converge
I am not sure if "converge" is a proper description of my problem. I'll state it in detail below.
I'm working on a CFD problem which means each sample is expensive. The framework I choose to ...
- 21
0
votes
0
answers
15
views
Periodic oscillations or discontinuities in likelihood function / loglikelihood function sampling over 2D grid of data
Is it expected behaviour that the loglikelihood function may have periodic oscillations within an envelope, or even periodic discontinuities for data sampled over a 2D grid?
For example, generate mock ...
- 1
1
vote
1
answer
102
views
Minimum sampling for maximising the prediction accuracy [closed]
Suppose that I'm training a machine learning model to predict people's age by a picture of their faces. Lets say that I have a dataset of people from 1 year olds to 100 year olds. But I want to choose ...
- 61
0
votes
0
answers
24
views
How to create a detection probability matrix for hypothesis testing
In Convex Optimization by Boyd there is a section on optimal detectors and hypothesis testing. I am having a hard time understanding how the randomized detectors work given the formulas.
The section ...
- 101
1
vote
0
answers
31
views
How to solve this type of multi-task Bayesian optimization problem?
Let us consider a collection of local Bayesian optimization tasks, each employs a Gaussian Process model to find the local optimum (i.e. global optimum of that task). The goal is to design a ...
- 173
0
votes
0
answers
17
views
How to evaluate likelihood in MCMC for arbitrarily shaped distributions?
I'm very confused with the use of MCMC to estimate distributions that have a complex shape, like multiple peaks, or that aren't generated from a known distribution. In particular, when calculating the ...
- 71
1
vote
0
answers
56
views
What is global concavity of the (log-)likelihood worth in Bayesian estimation?
In maximum likelihood estimation there is a big emphasis on finding the global maximum, which is why likelihood functions that are provably globally (log-)concave are desirable (despite often being ...
- 1,102
0
votes
0
answers
408
views
which dataset to send as eval set in xgboost, catboost, and etc, when using optuna
In some boost models there are option to send eval set while fitting the model.
for example:
...
- 301
2
votes
1
answer
45
views
Is it correct to replace the oldest data when using Gaussian Process in blackbox hyper-parameter optimization?
I try to apply GPR in a blackbox HPO question. My input will have 6 dimensions like X=[x1,...x6]. The implementation is quite straightforward with sklearn with a ...
- 145
0
votes
0
answers
57
views
How to restore Optuna's finished study from logs?
I wish to restore optuna.study.Study class object from uncomplete (5000 last lines, which is enough for my goals) logs. They look like that:
...
0
votes
0
answers
18
views
Using a kenel function which is different from the common set of kernels
The function $f:\mathbb{R}\rightarrow\mathbb{R}$ is a neural network that depends on $\theta$ (model parameters) and $\lambda$ (hyperparameters). Therefore, the surrogate function can be written as ...
- 101
1
vote
1
answer
473
views
How to run Bayesian optimization experiments in parallel?
Suppose I have the following hyperparameters for tuning:
learning_rate: [0.00001, 0.1]
epochs: [200,300,400,....,1000]
batch_size: [16,32,64,128]
If I want to run experiments using 4 parallel jobs, ...
- 887
1
vote
0
answers
28
views
How much reduction of hyperparameter experiments can I get using Bayesian optimization vs Grid search?
I have 5 hyperparameters for tuning and the number of combinations of all possible values is 9,360. This means if I want to find the optimal parameter setting using Grid Search, I need to do 9,360 ...
- 887
0
votes
0
answers
93
views
Why is it desirable to standardize the inputs of a Gaussian Process Regression?
I read this question (Should we standardize the data while doing Gaussian process regression?) and wondered, why do we need to normalize the inputs of a Gaussian Process?
In my case, I want to use ...
- 1
0
votes
0
answers
144
views
Implementation of bayesian optimization
Is there an easy to apply implemented tool in python or R for a bayesian optimization?
As I know this topic only superficially, I want to use bayesian optimization to determine the number of ...
- 2,899
2
votes
1
answer
22
views
Clarification on Bayesian ensembling
I was reading a paper https://arxiv.org/pdf/2007.06823.pdf and at the end of page 3 the author presents the technique called "ensembling" for the estimation of the expected outputs and the ...
- 259
0
votes
0
answers
131
views
what results are still usable from non-nested cross validation when tuning hyperparameters (and reconciling that with Optuna)
I'm trying to wrap my head around nested cross validation for the purposes of hyperparameter tuning.
Part i)
If I was to run hyperparameter tuning with cross validation (without nesting), say with ...
- 111
2
votes
1
answer
144
views
Is it possible to pick more than 1 sample point in each iteration of bayesian optimisation?
I want to use Bayesian optimisation for my project and I plan to build a closed-loop system, such that there is a model, robot to conduct experiments, measurement of experimental data which updates ...
- 41
3
votes
1
answer
102
views
How does removal of symmetry (e.g. via constraints) in a Bayesian optimization search space affect search efficiency?
There are many examples of search space symmetry in real-world optimization problems in the physical sciences. To motivate this, here are some that come to mind:
When optimizing a formulation such as ...
- 69
1
vote
1
answer
258
views
Package for hyperparameter optimization with categorical values
Context
I'm trying to solve a black-box optimization problem, and I can "reformulate" parts of the problem is different ways that may lead to lower or higher costs, and which can interact ...
- 97
1
vote
0
answers
54
views
Use of Monte Carlo Tree Search
I was talking with someone much more experienced in stats than I am and they suggested the use of Monte Carlo Tree Search for a problem I am facing.
Problem Statement: I am collecting jitter ...
- 21
1
vote
1
answer
71
views
What is the difference between Bayesian Optimizaiton for hyperparameters and using validation set when training?
While studying hyperparameter tuning in Machine Learning, I have come to read Bayesian Optimization for Hyperparameter Tuning and Using validation set when training the model but it is kind of ...
- 2,695
0
votes
0
answers
249
views
Bayesian optimization resulting in overfitting despite split test train
I am using XGBRegressor on a time series to predict the next periods value. I use sklearns TimeSeriesSplit to split the data into test and train.
I use skopts BayesSearchCV to search through a ...
0
votes
2
answers
135
views
Is the result of Bayesian Inference with MCMC reliable since there maybe a big variance?
I'm new to Bayesian inference and I came across a, maybe simple, question. That is, in many cases, we don't know how the posterior distribution really looks like, and we adapt the result of MCMC, or ...
- 101
1
vote
1
answer
179
views
Gaussian process on hyperparameter tuning
I feel it is kind of circular to use GP for hyperparameter tuning, since GP has its own hyperparameters. Or is it the case that GP typically has less number of hyperparameters than the model we want ...
- 273
0
votes
0
answers
29
views
Why Bayesian optimization in its simplest form only works for low dimensional optimization? [duplicate]
This is what I learned from the class on how posterior is updated for Gaussian process:
It seems the computational cost of the matrix inversion only scales with the size of K_t, which depends on the ...
- 273
0
votes
0
answers
56
views
Optimizing over a subset of input variables
Consider the unconstraint optimization problem:
$$\arg \min_{𝑋_0} f(X_0, X_\eta)$$
where
$X_0$ is the input vector I need to optimize for
$𝑋_\eta$ is a nuisance vector which I do not care about.
...
- 101
1
vote
1
answer
92
views
How to implement Bayesian Optimization when there is a relationship between search space parameters?
I'm quite new to Bayesian models. I'm trying to implement Bayesian Optimization where the sum of parameters should be less than a constant.
Say, the parameters are ...
8
votes
2
answers
574
views
Question of understanding regarding Bayesian Optimization, Gaussian process and acquisition function
I'm trying to understand Bayesian optimization and I struggle a lot with all the involved methods. Hence, I have some short questions:
We start with a a-prior function, which is a gaussian process.
A ...
- 2,899
1
vote
0
answers
52
views
Using a different (but related) hypothesis for the prior in MAP
Say we have a general set of data $\mathcal{D} = \{\mathbf{x}_i, \mathbf{y}_i \}_{i \in N}$ of covariates $\mathbf{x}$ and observations $\mathbf{y}$. Our problem is in fitting a known model $\mathbf{y}...
- 11
1
vote
0
answers
79
views
Gaussian Process for noise free data
In a Gaussian Process model, the covariance function is given as follows:
$$
C\left(y_{*} \mid y_{d}\right)=K\left(x_{*}, x_{*}\right)-K\left(x_{*}, x_{d}\right) K\left(x_{d}, x_{d}\right)^{-1} K\left(...
- 11
0
votes
1
answer
128
views
Optimizing EI in the TPE algorithm
My question refers to the section "4.1 Optimizing EI in the TPE algorithm" on page 4 of the paper Algorithms for Hyperparameter Optimization (https://papers.nips.cc/paper/2011/file/...
- 161
1
vote
0
answers
62
views
I am confused about the sum indices of these posterior distribution formulas
I am reading this notes and trying to understand how the posterior distribution formulas of the variational variables involved have been calculated. I am confused about the indices of the summation. ...
- 176
1
vote
1
answer
290
views
Bayesian optimization - One run with 100 iterations vs ten runs with ten iterations
I was wondering whether it makes more sense to run Bayesian Optimization let's say ones with 100 iterations or ten times with ten iterations each. Which one should I favor?
I would assume that the ...
- 533
0
votes
0
answers
110
views
why Gamma inverse is the conjugate prior of normal distribution?
I am trying to understand Bayesian regression. Then in Wikipedia enter link description here, it is written that by using the following relation
we get the Gamma inverse as follow for the conjugate ...
- 143
0
votes
1
answer
107
views
Which kinds of priors could be applied in Bayesian linear regression?
I would like to know can we use any kind of prior distribution in Bayesian linear regression and still convergent to a close mean solution to the Least square solution? Or it should be just a ...
- 143
3
votes
1
answer
708
views
Bayesian Optimization vs. gradient descent
I don't know if this is the right place to ask this question. If you think this question is better asked in another StackExchange, please point me to that.
This question is about the sampling ...
- 171