Questions tagged [bayesian-optimization]

Bayesian optimization is a family of global optimization methods which use information about previously-computed values of the function to make inference about which function values are plausibly optima. Its applications include computer experiments and hyper-parameter optimization in some machine learning models.

Filter by
Sorted by
Tagged with
1 vote
1 answer
25 views

What exactly are we training across different iterations in the Gaussian Process Regression example in GPyTorch?

I am following this tutorial to implement a GP Regression using gPyTorch. Based on my understanding of GP Regression, given the training data we can compute the posterior mean and covariance using the ...
Namit Juneja's user avatar
2 votes
0 answers
47 views

Bayesian optimization for parametric curve fitting?

I am relatively new to Gaussian Processes and Bayesian Optimization. My question is very simple: Suppose I am trying to learn a function from a parametric family of curves which best describes the ...
chesslad's user avatar
3 votes
2 answers
135 views

How to choose a point that has both optimal value and low variance

I have a Gaussian Process Regression model that models the cost of a certain process. Once trained, I want to find the point $x$ corresponding to which the regression predicts the lowest cost. Simply ...
Namit Juneja's user avatar
2 votes
1 answer
112 views

Bayesian optimization for solving least squares

Bayesian optimization with Gaussian processes (GPs) is an effective minimization methodology when the evaluation of the function to minimize, say $f(a)$, is computationally expensive. Loosely speaking,...
altroware's user avatar
  • 206
0 votes
0 answers
11 views

A surrogate function for validation error in order to perform hyper-parameter optimization?

Greed search CV or few other approaches may be computationally expensive in hyper-parameter tuning. Is it possible to come up with a surrogate model or a purposed model for a validation error in order ...
Lakshman's user avatar
3 votes
0 answers
108 views

Bayesian Optimization: number of iterations as function of search space dimensionality?

I am performing Bayesian Optimization to select a hyperparameter configuration for my supervised learning model. I understand that with each additional hyperparameter that I choose to optimize, the ...
DavidSilverberg's user avatar
1 vote
0 answers
39 views

Rounding Approximation for Blackbox Integer Optimization

I am working on a black-box optimization that involves surrogate modeling. Some of my decision variables are integers, but I doubt a MIP approach would work for my case. My advisor told me that it is ...
Kaiwen Wang's user avatar
1 vote
0 answers
25 views

Expected improvement for bayesian linear regression with unknown noise variance

My question is basically if the expected improvement for a bayesian linear regression with unknown noise variance, i.e. we place a prior on the noise variance -> predictive distribution may not be ...
optimalic's user avatar
  • 111
11 votes
2 answers
1k views

Why go through the trouble of expectation maximization and not use gradient descent?

In expectation maximization first a lower bound of the likelihood is found and then a 2 step iterative algorithm kicks in where first we try to find the weights (the probability that a data point ...
figs_and_nuts's user avatar
0 votes
0 answers
53 views

Tuning Random Forest results in max_features parameter taking a value of 1. Why?

I did a bayesian optimization tuning for parameters of random forest. With 200 iterations, it seems like 70% of the times, very low values (read 1 or 2) of max_features seems to produce better (...
MSKO's user avatar
  • 61
0 votes
0 answers
19 views

Calculate acceptance ratio of Jacobian of split-merge RJMCMC

I am keep studying the RJMCMC and want to ask question regarding the acceptance ratio of split/merge step of RJMCMC The split/merge step, suggested by Richardson and Green (1997) is following for w_j, ...
Kyungmin's user avatar
4 votes
1 answer
196 views

Terms and assumptions in trans-dimensional MCMC (RJ-MCMC) for Green 1995 paper

I want to use Trans-dimensional MCMC in my research and for fundamental understanding, I am trying to learn from Green (1995) paper, which is foundation of RJ-MCMC. In part of 3.3 'switching between ...
Kyungmin's user avatar
1 vote
0 answers
23 views

With mult-dimensional input vectors, what are the dimensions of the covariance matrix elements? (Gaussian Process)

I am trying to create a Bayesian Optimisation code with a Gaussian Process. My input data, $\vec{X}_i$ is 8-dimensional, where each dimension corresponds to a feature of my data, $\vec{X}_i = [\...
shadowbiscuit's user avatar
0 votes
0 answers
32 views

Log likelihood decreases with posterior obtained after fitting GP

Possibly related to Log posterior probability in MCMC is decreasing but I do not have a MCMC process and the details there are not sufficient for me to understand fully (I'm a mathematician with basic ...
F. Remonato's user avatar
0 votes
0 answers
33 views

Hyperparameter tuning and initialisation doubts in multivariate gaussian process model

I'm trying to train a multivariate Gaussian Process model using the code here https://github.com/Magica-Chen/gptp_multi_output. However I noticed how problematic is to initialise the length scales of ...
Roldes's user avatar
  • 1
0 votes
0 answers
12 views

Applying EI to GP-based multi-fidelity modeling

I am working on optimizing a GP-based multi-fidelity surrogate model and I found this publication very helpful: 'A Tutorial on Bayesian Optimization' by Peter I. Frazier (July 10, 2018). and I have ...
Ann's user avatar
  • 43
6 votes
2 answers
819 views

Estimating a "most likely" distribution from min, max, mean, median, standard deviation

I'm a physics undergrad who started becoming curious about this question after exam season. After any exam, we're typically given the following parameters: Min, max, mean, median, std. deviation, ...
SettingStar's user avatar
1 vote
1 answer
379 views

Difference between Bayesian optimization and multi-armed bandit optimization

What are the differences between Bayesian optimization and multi-armed bandit optimization? Are the problems equivalent when multi-armed bandit's action space is infinite?
fool's user avatar
  • 2,440
0 votes
0 answers
35 views

Best function value for Expected Improvement computation for Gaussian Process Regression

I want to implement Gaussian Process regression in the context of active learning, in which interpolation is performed with the best interpolating points, selected at each step iteratively. At every ...
Andrea Gulli's user avatar
0 votes
0 answers
89 views

How to combine Bayesian Optimization and Cross Validation

I am currently using a train-validation-test split and Bayesian Optimization (BO), which is straightforward. Now, I want to transition to using 3-fold cross validation and BO (+an additional test set ...
Enk9456's user avatar
  • 33
1 vote
0 answers
56 views

Best way to fit a Gaussian Process surrogate model to an RL Reward function

Is there a way to get an estimate of good scaling parameters (namely mean and variance) for a Gaussian Process kernel serving as a surrogate model to a Reinforcement Learning reward function for ...
Prishita Ray's user avatar
2 votes
0 answers
19 views

Constrained optimization between two bayesian variables

I have 2 separate Bayesain networks and I was hoping to maximize Value within the constraint of the Cost. What are is a good way ...
stat_math's user avatar
5 votes
1 answer
215 views

Deriving Log Marginal Likelihood for Gaussian Process

I am trying to evaluate the following integral marginalized across all possible functions. $$\mathbb{P}(y|X,\theta) = \int \mathbb{P}(y|f)\ \mathbb{P}(f|X,\theta) \ df$$ In G.P. we assume prior to be ...
Black Beard 53's user avatar
3 votes
1 answer
160 views

Bayesian optimization on a polynomial regression as the surrogate model

My understanding of Bayesian optimization is that it is generally used in conjunction with Gaussian process (GP) as the surrogate model. Because GP inherently produces an uncertainty of estimate, the ...
Tianxun Zhou's user avatar
1 vote
0 answers
82 views

Monte Carlo Dropout as surrogate model for Bayesian Optimization

I am interested in using Monte Carlo Dropout as a surrogate model for Bayesian optimization. I noticed that the paper states: The use of dropout (and its variants) in NNs can be interpreted as a ...
Eismont's user avatar
  • 11
3 votes
1 answer
285 views

Role of standard deviation in Bayesian optimization using GP

I am new to GP and BO and I have been playing with the two in a simple 1D context which happens to be practically relevant to what I am working on. Essentially, I am trying to find a peak (modeled as ...
jrmxn's user avatar
  • 53
1 vote
1 answer
36 views

Bayesian terminology for experiments

I am learning about bayesian experimental design and confused about the "Bayesian" terminology. Multi-armed bandits are normally in the syllabus of bayesian experiments. But there are ...
user2715182's user avatar
4 votes
0 answers
262 views

Choice of model surrogate for bayesian optimization

I am running BayesSearchCV to optimize the hyperparameters of my machine learning model. This particular procedure allows the user to choose the surrogate model. The options are Gaussian Process, ...
DavidSilverberg's user avatar
0 votes
1 answer
62 views

Bayesian optimization with constraints

I want to perform Bayesian optimization for a certain physical task but with additional requirements. We have access to a set of variables and want to maximize (multiple) signal outputs from an ...
arod's user avatar
  • 23
1 vote
0 answers
75 views

Why was Bayes better on exponential model rather than log-linear model?

I wanted to train a Bayesian version of this model which we can consider to be this log-linear form. $$\ln \text{PSI} = \alpha \text{Time} + \beta.$$ Here are the priors I guessed for $\alpha$ and $\...
Galen's user avatar
  • 8,145
1 vote
0 answers
49 views

How to apply Bayesian Optimization to a function depending only on categorical variables?

I would like to apply Bayesian Optimization (BO) to a black-box function depending only on multiple categorical variables. In my application, each categorical variable has 3 possible categories. I ...
CharlelieLrt's user avatar
0 votes
0 answers
168 views

Expected Improvement (EI) in Bayesian Optimization failed to converge

I am not sure if "converge" is a proper description of my problem. I'll state it in detail below. I'm working on a CFD problem which means each sample is expensive. The framework I choose to ...
Ann's user avatar
  • 43
1 vote
1 answer
107 views

Minimum sampling for maximising the prediction accuracy [closed]

Suppose that I'm training a machine learning model to predict people's age by a picture of their faces. Lets say that I have a dataset of people from 1 year olds to 100 year olds. But I want to choose ...
noone's user avatar
  • 73
1 vote
0 answers
40 views

How to solve this type of multi-task Bayesian optimization problem?

Let us consider a collection of local Bayesian optimization tasks, each employs a Gaussian Process model to find the local optimum (i.e. global optimum of that task). The goal is to design a ...
Shaun Han's user avatar
  • 173
0 votes
0 answers
21 views

How to evaluate likelihood in MCMC for arbitrarily shaped distributions?

I'm very confused with the use of MCMC to estimate distributions that have a complex shape, like multiple peaks, or that aren't generated from a known distribution. In particular, when calculating the ...
questionmark's user avatar
1 vote
0 answers
139 views
+100

What is global concavity of the (log-)likelihood worth in Bayesian estimation?

In maximum likelihood estimation there is a big emphasis on finding the global maximum, which is why likelihood functions that are provably globally (log-)concave are desirable (despite often being ...
Durden's user avatar
  • 1,175
2 votes
1 answer
56 views

Is it correct to replace the oldest data when using Gaussian Process in blackbox hyper-parameter optimization?

I try to apply GPR in a blackbox HPO question. My input will have 6 dimensions like X=[x1,...x6]. The implementation is quite straightforward with sklearn with a ...
G_cy's user avatar
  • 145
1 vote
2 answers
1k views

How to run Bayesian optimization experiments in parallel?

Suppose I have the following hyperparameters for tuning: learning_rate: [0.00001, 0.1] epochs: [200,300,400,....,1000] batch_size: [16,32,64,128] If I want to run experiments using 4 parallel jobs, ...
etang's user avatar
  • 907
1 vote
0 answers
45 views

How much reduction of hyperparameter experiments can I get using Bayesian optimization vs Grid search?

I have 5 hyperparameters for tuning and the number of combinations of all possible values is 9,360. This means if I want to find the optimal parameter setting using Grid Search, I need to do 9,360 ...
etang's user avatar
  • 907
0 votes
0 answers
247 views

Why is it desirable to standardize the inputs of a Gaussian Process Regression?

I read this question (Should we standardize the data while doing Gaussian process regression?) and wondered, why do we need to normalize the inputs of a Gaussian Process? In my case, I want to use ...
robl97's user avatar
  • 1
0 votes
0 answers
222 views

Implementation of bayesian optimization

Is there an easy to apply implemented tool in python or R for a bayesian optimization? As I know this topic only superficially, I want to use bayesian optimization to determine the number of ...
Ben's user avatar
  • 3,353
2 votes
1 answer
23 views

Clarification on Bayesian ensembling

I was reading a paper https://arxiv.org/pdf/2007.06823.pdf and at the end of page 3 the author presents the technique called "ensembling" for the estimation of the expected outputs and the ...
Alucard's user avatar
  • 325
2 votes
1 answer
317 views

Is it possible to pick more than 1 sample point in each iteration of bayesian optimisation?

I want to use Bayesian optimisation for my project and I plan to build a closed-loop system, such that there is a model, robot to conduct experiments, measurement of experimental data which updates ...
wowewow's user avatar
  • 51
3 votes
1 answer
155 views

How does removal of symmetry (e.g. via constraints) in a Bayesian optimization search space affect search efficiency?

There are many examples of search space symmetry in real-world optimization problems in the physical sciences. To motivate this, here are some that come to mind: When optimizing a formulation such as ...
Sterling's user avatar
2 votes
1 answer
583 views

Package for hyperparameter optimization with categorical values

Context I'm trying to solve a black-box optimization problem, and I can "reformulate" parts of the problem is different ways that may lead to lower or higher costs, and which can interact ...
swineone's user avatar
1 vote
0 answers
65 views

Use of Monte Carlo Tree Search

I was talking with someone much more experienced in stats than I am and they suggested the use of Monte Carlo Tree Search for a problem I am facing. Problem Statement: I am collecting jitter ...
Paul Kumar's user avatar
2 votes
1 answer
140 views

What is the difference between Bayesian Optimizaiton for hyperparameters and using validation set when training?

While studying hyperparameter tuning in Machine Learning, I have come to read Bayesian Optimization for Hyperparameter Tuning and Using validation set when training the model but it is kind of ...
xabzakabecd's user avatar
  • 3,315
0 votes
0 answers
443 views

Bayesian optimization resulting in overfitting despite split test train

I am using XGBRegressor on a time series to predict the next periods value. I use sklearns TimeSeriesSplit to split the data into test and train. I use skopts BayesSearchCV to search through a ...
Matthias Tom's user avatar
0 votes
2 answers
309 views

Is the result of Bayesian Inference with MCMC reliable since there maybe a big variance?

I'm new to Bayesian inference and I came across a, maybe simple, question. That is, in many cases, we don't know how the posterior distribution really looks like, and we adapt the result of MCMC, or ...
C.K.'s user avatar
  • 131
2 votes
1 answer
265 views

Gaussian process on hyperparameter tuning

I feel it is kind of circular to use GP for hyperparameter tuning, since GP has its own hyperparameters. Or is it the case that GP typically has less number of hyperparameters than the model we want ...
Sam's user avatar
  • 363