# Questions tagged [bayesian-optimization]

Bayesian optimization is a family of global optimization methods which use information about previously-computed values of the function to make inference about which function values are plausibly optima. Its applications include computer experiments and hyper-parameter optimization in some machine learning models.

171 questions
Filter by
Sorted by
Tagged with
15 views

### Periodic oscillations or discontinuities in likelihood function / loglikelihood function sampling over 2D grid of data

Is it expected behaviour that the loglikelihood function may have periodic oscillations within an envelope, or even periodic discontinuities for data sampled over a 2D grid? For example, generate mock ...
1 vote
97 views

### Minimum sampling for maximising the prediction accuracy [closed]

Suppose that I'm training a machine learning model to predict people's age by a picture of their faces. Lets say that I have a dataset of people from 1 year olds to 100 year olds. But I want to choose ...
• 61
18 views

### How to create a detection probability matrix for hypothesis testing

In Convex Optimization by Boyd there is a section on optimal detectors and hypothesis testing. I am having a hard time understanding how the randomized detectors work given the formulas. The section ...
• 101
1 vote
23 views

### How to solve this type of multi-task Bayesian optimization problem?

Let us consider a collection of local Bayesian optimization tasks, each employs a Gaussian Process model to find the local optimum (i.e. global optimum of that task). The goal is to design a ...
• 163
12 views

### How to evaluate likelihood in MCMC for arbitrarily shaped distributions?

I'm very confused with the use of MCMC to estimate distributions that have a complex shape, like multiple peaks, or that aren't generated from a known distribution. In particular, when calculating the ...
1 vote
32 views

### What is global concavity of the (log-)likelihood worth in Bayesian estimation?

In maximum likelihood estimation there is a big emphasis on finding the global maximum, which is why likelihood functions that are provably globally (log-)concave are desirable (despite often being ...
• 1,000
126 views

### which dataset to send as eval set in xgboost, catboost, and etc, when using optuna

In some boost models there are option to send eval set while fitting the model. for example: ...
• 77
34 views

### Is it correct to replace the oldest data when using Gaussian Process in blackbox hyper-parameter optimization?

I try to apply GPR in a blackbox HPO question. My input will have 6 dimensions like X=[x1,...x6]. The implementation is quite straightforward with sklearn with a ...
• 145
32 views

### How to restore Optuna's finished study from logs?

I wish to restore optuna.study.Study class object from uncomplete (5000 last lines, which is enough for my goals) logs. They look like that: ...
15 views

### Using a kenel function which is different from the common set of kernels

The function $f:\mathbb{R}\rightarrow\mathbb{R}$ is a neural network that depends on $\theta$ (model parameters) and $\lambda$ (hyperparameters). Therefore, the surrogate function can be written as ...
• 101
1 vote
165 views

### How to run Bayesian optimization experiments in parallel?

Suppose I have the following hyperparameters for tuning: learning_rate: [0.00001, 0.1] epochs: [200,300,400,....,1000] batch_size: [16,32,64,128] If I want to run experiments using 4 parallel jobs, ...
• 867
1 vote
23 views

### How much reduction of hyperparameter experiments can I get using Bayesian optimization vs Grid search?

I have 5 hyperparameters for tuning and the number of combinations of all possible values is 9,360. This means if I want to find the optimal parameter setting using Grid Search, I need to do 9,360 ...
• 867
44 views

### Why is it desirable to standardize the inputs of a Gaussian Process Regression?

I read this question (Should we standardize the data while doing Gaussian process regression?) and wondered, why do we need to normalize the inputs of a Gaussian Process? In my case, I want to use ...
72 views

### Implementation of bayesian optimization

Is there an easy to apply implemented tool in python or R for a bayesian optimization? As I know this topic only superficially, I want to use bayesian optimization to determine the number of ...
• 2,441
22 views

### Clarification on Bayesian ensembling

I was reading a paper https://arxiv.org/pdf/2007.06823.pdf and at the end of page 3 the author presents the technique called "ensembling" for the estimation of the expected outputs and the ...
• 207
62 views

### what results are still usable from non-nested cross validation when tuning hyperparameters (and reconciling that with Optuna)

I'm trying to wrap my head around nested cross validation for the purposes of hyperparameter tuning. Part i) If I was to run hyperparameter tuning with cross validation (without nesting), say with ...
• 111
1 vote
94 views

### Is it possible to pick more than 1 sample point in each iteration of bayesian optimisation?

I want to use Bayesian optimisation for my project and I plan to build a closed-loop system, such that there is a model, robot to conduct experiments, measurement of experimental data which updates ...
• 31
82 views

### How does removal of symmetry (e.g. via constraints) in a Bayesian optimization search space affect search efficiency?

There are many examples of search space symmetry in real-world optimization problems in the physical sciences. To motivate this, here are some that come to mind: When optimizing a formulation such as ...
• 69
1 vote
120 views

### Package for hyperparameter optimization with categorical values

Context I'm trying to solve a black-box optimization problem, and I can "reformulate" parts of the problem is different ways that may lead to lower or higher costs, and which can interact ...
• 97
1 vote
30 views

### Use of Monte Carlo Tree Search

I was talking with someone much more experienced in stats than I am and they suggested the use of Monte Carlo Tree Search for a problem I am facing. Problem Statement: I am collecting jitter ...
36 views

### Fixed-leg Kalman filter smoother (Rauch–Tung–Striebel) error bounds

Although very intuitive and with plenty of results that talk about the asymptotic convergence of the estimate I wasn't able to track down any paper stating explicitly convergence bounds based on the ...
• 1
18 views

### Discussion of hyperparamter to optimize

For my thesis I consider to add a small discussion about hyperparameter optimization (HPO). I have five tree based models I would like to tune. Namely RandomForest, GBM, XGBoost, LightGBM and CatBoost....
• 218
1 vote
42 views

### What is the difference between Bayesian Optimizaiton for hyperparameters and using validation set when training?

While studying hyperparameter tuning in Machine Learning, I have come to read Bayesian Optimization for Hyperparameter Tuning and Using validation set when training the model but it is kind of ...
• 2,655
151 views

### Bayesian optimization resulting in overfitting despite split test train

I am using XGBRegressor on a time series to predict the next periods value. I use sklearns TimeSeriesSplit to split the data into test and train. I use skopts BayesSearchCV to search through a ...
80 views

### Is the result of Bayesian Inference with MCMC reliable since there maybe a big variance?

I'm new to Bayesian inference and I came across a, maybe simple, question. That is, in many cases, we don't know how the posterior distribution really looks like, and we adapt the result of MCMC, or ...
• 101
34 views

### How to solve optimization problem?

I am trying to implement the following from a paper I am reading. I have reached the following part, but I am unsure of how to solve the optimization problem. The authors do not specify what method ...
1 vote
101 views

### Gaussian process on hyperparameter tuning

I feel it is kind of circular to use GP for hyperparameter tuning, since GP has its own hyperparameters. Or is it the case that GP typically has less number of hyperparameters than the model we want ...
• 223
27 views

### Why Bayesian optimization in its simplest form only works for low dimensional optimization? [duplicate]

This is what I learned from the class on how posterior is updated for Gaussian process: It seems the computational cost of the matrix inversion only scales with the size of K_t, which depends on the ...
• 223
67 views

### Expected Improvement for Sequential Model-based Global Optimization

I am reading a paper about hyper-parameter optimization (Bergstra et al., 2011), and I have a question about the expected improvement. In the paper, the pseudo-code of a generic SMBO is shown as ...
33 views

### Optimizing over a subset of input variables

Consider the unconstraint optimization problem: $$\arg \min_{𝑋_0} f(X_0, X_\eta)$$ where $X_0$ is the input vector I need to optimize for $𝑋_\eta$ is a nuisance vector which I do not care about. ...
• 101
1 vote
60 views

### How to implement Bayesian Optimization when there is a relationship between search space parameters?

I'm quite new to Bayesian models. I'm trying to implement Bayesian Optimization where the sum of parameters should be less than a constant. Say, the parameters are ...
418 views

### Question of understanding regarding Bayesian Optimization, Gaussian process and acquisition function

I'm trying to understand Bayesian optimization and I struggle a lot with all the involved methods. Hence, I have some short questions: We start with a a-prior function, which is a gaussian process. A ...
• 2,441
1 vote
52 views

• 651
395 views

### Question regarding log marginal likelihood in SKLearn

I'm trying to understand the hyperparameter optimization implemented in SKLearn. I'm using the basic example presented here with an alternative data set of 100 observations of Rastrigin test function (...
• 1
73 views

### Is it valid to perform Bayesian optimization and MCMC sampling with nested model parameters?

I am trying to think of a way to perform a somewhat complicated Bayesian optimization problem involving model parameters which are nested within other functions in the forward problem, and I am ...
• 845
1 vote
46 views

### Bayesian Optimization with transformed objective function

I have a sequential learning problem where I want to rank a group of jobs in each iteration--unlike conventional Bayesian approach, I am trying to find the ordering of jobs based on g(f(x)) where GP ...
• 11
758 views

### How does Bayesian Optimization balance exploration with exploitation?

I'm decently familiar with Gaussian Processes; I understand that GPs form the backbone of BO but I don't want this question to drift in scope towards an explanation of GPs. Rather, I'm curious, how ...
• 1,728
1 vote