Questions tagged [bayesian-optimization]

Bayesian optimization is a family of global optimization methods which use information about previously-computed values of the function to make inference about which function values are plausibly optima. Its applications include computer experiments and hyper-parameter optimization in some machine learning models.

Filter by
Sorted by
Tagged with
0 votes
0 answers
5 views

Why is it desirable to standardize the inputs of a Gaussian Process Regression?

I read this question (Should we standardize the data while doing Gaussian process regression?) and wondered, why do we need to normalize the inputs of a Gaussian Process? In my case, I want to use ...
user avatar
  • 1
0 votes
0 answers
21 views

Implementation of bayesian optimization

Is there an easy to apply implemented tool in python or R for a bayesian optimization? As I know this topic only superficially, I want to use bayesian optimization to determine the number of ...
user avatar
  • 2,187
1 vote
1 answer
21 views

Clarification on Bayesian ensembling

I was reading a paper https://arxiv.org/pdf/2007.06823.pdf and at the end of page 3 the author presents the technique called "ensembling" for the estimation of the expected outputs and the ...
user avatar
  • 113
0 votes
0 answers
15 views

what results are still usable from non-nested cross validation when tuning hyperparameters (and reconciling that with Optuna)

I'm trying to wrap my head around nested cross validation for the purposes of hyperparameter tuning. Part i) If I was to run hyperparameter tuning with cross validation (without nesting), say with ...
user avatar
  • 101
1 vote
1 answer
39 views

Is it possible to pick more than 1 sample point in each iteration of bayesian optimisation?

I want to use Bayesian optimisation for my project and I plan to build a closed-loop system, such that there is a model, robot to conduct experiments, measurement of experimental data which updates ...
user avatar
  • 21
3 votes
1 answer
71 views

How does removal of symmetry (e.g. via constraints) in a Bayesian optimization search space affect search efficiency?

There are many examples of search space symmetry in real-world optimization problems in the physical sciences. To motivate this, here are some that come to mind: When optimizing a formulation such as ...
user avatar
1 vote
1 answer
38 views

Package for hyperparameter optimization with categorical values

Context I'm trying to solve a black-box optimization problem, and I can "reformulate" parts of the problem is different ways that may lead to lower or higher costs, and which can interact ...
user avatar
1 vote
0 answers
17 views

Use of Monte Carlo Tree Search

I was talking with someone much more experienced in stats than I am and they suggested the use of Monte Carlo Tree Search for a problem I am facing. Problem Statement: I am collecting jitter ...
user avatar
0 votes
0 answers
16 views

Fixed-leg Kalman filter smoother (Rauch–Tung–Striebel) error bounds

Although very intuitive and with plenty of results that talk about the asymptotic convergence of the estimate I wasn't able to track down any paper stating explicitly convergence bounds based on the ...
user avatar
  • 1
0 votes
0 answers
17 views

Discussion of hyperparamter to optimize

For my thesis I consider to add a small discussion about hyperparameter optimization (HPO). I have five tree based models I would like to tune. Namely RandomForest, GBM, XGBoost, LightGBM and CatBoost....
user avatar
  • 218
1 vote
1 answer
27 views

What is the difference between Bayesian Optimizaiton for hyperparameters and using validation set when training?

While studying hyperparameter tuning in Machine Learning, I have come to read Bayesian Optimization for Hyperparameter Tuning and Using validation set when training the model but it is kind of ...
user avatar
  • 2,585
0 votes
0 answers
80 views

Bayesian optimization resulting in overfitting despite split test train

I am using XGBRegressor on a time series to predict the next periods value. I use sklearns TimeSeriesSplit to split the data into test and train. I use skopts BayesSearchCV to search through a ...
user avatar
0 votes
2 answers
57 views

Is the result of Bayesian Inference with MCMC reliable since there maybe a big variance?

I'm new to Bayesian inference and I came across a, maybe simple, question. That is, in many cases, we don't know how the posterior distribution really looks like, and we adapt the result of MCMC, or ...
user avatar
  • 101
0 votes
0 answers
34 views

How to solve optimization problem?

I am trying to implement the following from a paper I am reading. I have reached the following part, but I am unsure of how to solve the optimization problem. The authors do not specify what method ...
user avatar
1 vote
1 answer
57 views

Gaussian process on hyperparameter tuning

I feel it is kind of circular to use GP for hyperparameter tuning, since GP has its own hyperparameters. Or is it the case that GP typically has less number of hyperparameters than the model we want ...
user avatar
  • 223
0 votes
0 answers
27 views

Why Bayesian optimization in its simplest form only works for low dimensional optimization? [duplicate]

This is what I learned from the class on how posterior is updated for Gaussian process: It seems the computational cost of the matrix inversion only scales with the size of K_t, which depends on the ...
user avatar
  • 223
0 votes
0 answers
38 views

Expected Improvement for Sequential Model-based Global Optimization

I am reading a paper about hyper-parameter optimization (Bergstra et al., 2011), and I have a question about the expected improvement. In the paper, the pseudo-code of a generic SMBO is shown as ...
user avatar
0 votes
0 answers
25 views

Optimizing over a subset of input variables

Consider the unconstraint optimization problem: $$\arg \min_{𝑋_0} f(X_0, X_\eta)$$ where $X_0$ is the input vector I need to optimize for $𝑋_\eta$ is a nuisance vector which I do not care about. ...
user avatar
  • 101
1 vote
1 answer
55 views

How to implement Bayesian Optimization when there is a relationship between search space parameters?

I'm quite new to Bayesian models. I'm trying to implement Bayesian Optimization where the sum of parameters should be less than a constant. Say, the parameters are ...
user avatar
6 votes
2 answers
304 views

Question of understanding regarding Bayesian Optimization, Gaussian process and acquisition function

I'm trying to understand Bayesian optimization and I struggle a lot with all the involved methods. Hence, I have some short questions: We start with a a-prior function, which is a gaussian process. A ...
user avatar
  • 2,187
0 votes
0 answers
10 views

How to allocate the number of samples depending on the variances

I want to compute $p$ number of independent functions with each giving their result as expectation value, $e_{i=1,2..p}$. The final result is the sum of all expectation values, $S = \sum_{i}^{p}e_{i}$....
user avatar
1 vote
0 answers
52 views

Using a different (but related) hypothesis for the prior in MAP

Say we have a general set of data $\mathcal{D} = \{\mathbf{x}_i, \mathbf{y}_i \}_{i \in N}$ of covariates $\mathbf{x}$ and observations $\mathbf{y}$. Our problem is in fitting a known model $\mathbf{y}...
user avatar
1 vote
0 answers
38 views

Gaussian Process for noise free data

In a Gaussian Process model, the covariance function is given as follows: $$ C\left(y_{*} \mid y_{d}\right)=K\left(x_{*}, x_{*}\right)-K\left(x_{*}, x_{d}\right) K\left(x_{d}, x_{d}\right)^{-1} K\left(...
user avatar
0 votes
1 answer
43 views

Optimizing EI in the TPE algorithm

My question refers to the section "4.1 Optimizing EI in the TPE algorithm" on page 4 of the paper Algorithms for Hyperparameter Optimization (https://papers.nips.cc/paper/2011/file/...
user avatar
  • 151
1 vote
0 answers
58 views

I am confused about the sum indices of these posterior distribution formulas

I am reading this notes and trying to understand how the posterior distribution formulas of the variational variables involved have been calculated. I am confused about the indices of the summation. ...
user avatar
0 votes
0 answers
42 views

What is the relationship between Bayesian Optimization and Sequential Model Based Optimization?

I often see these two terms being used almost interchangeably: Bayesian Optimization (BO) and Sequential Model Based Optimization (SMBO). Sometimes, SMBO is referred to as a formalisation/formalism of ...
user avatar
  • 151
1 vote
1 answer
58 views

Bayesian optimization - One run with 100 iterations vs ten runs with ten iterations

I was wondering whether it makes more sense to run Bayesian Optimization let's say ones with 100 iterations or ten times with ten iterations each. Which one should I favor? I would assume that the ...
user avatar
  • 533
0 votes
0 answers
32 views

why Gamma inverse is the conjugate prior of normal distribution?

I am trying to understand Bayesian regression. Then in Wikipedia enter link description here, it is written that by using the following relation we get the Gamma inverse as follow for the conjugate ...
user avatar
  • 123
0 votes
1 answer
44 views

Which kinds of priors could be applied in Bayesian linear regression?

I would like to know can we use any kind of prior distribution in Bayesian linear regression and still convergent to a close mean solution to the Least square solution? Or it should be just a ...
user avatar
  • 123
3 votes
1 answer
258 views

Bayesian Optimization vs. gradient descent

I don't know if this is the right place to ask this question. If you think this question is better asked in another StackExchange, please point me to that. This question is about the sampling ...
user avatar
  • 161
0 votes
0 answers
14 views

Data acquisition for subsequent global/bayesian optimisation

For a research project, we are interested in applying global/bayesian optimisation for engineering an enzyme for optimal/maximum efficiency. The assumption is that there is a functional relationship $...
user avatar
0 votes
0 answers
17 views

Can't accurately estimate decay-fraction parameter from simulation using JAGS in R

I am generating simulated data of exponential decay of a variable over 10 sessions for 5 subjects, and seeing if I can estimate the 2 parameters (starting value and fraction of decay for each ...
user avatar
1 vote
1 answer
136 views

What distributions to use for hyperparameter tuning?

Where would one start when trying to figure out which distributions to use for hyperparameter tuning? Libraries such as HyperOpt, Optuna, and sklearn (random search) ask not for uniformly distributed ...
user avatar
  • 113
1 vote
1 answer
36 views

Bayesian optmization when the relationship on some variables linear, and non-linear on other features

I am working on a Bayesian Optimization problem, where the approximator is GP, and the utility function is UCB1. My problem is the output of the function is linear on some features while non-linear on ...
user avatar
1 vote
0 answers
57 views

Can I use AdaBoost Regressor with Gaussian Process Regressor as base estimator, as my Surrogate Function?

I am using a text-extraction model whose hyperparamters I want to optimise. The model takes time(on average 1 hr) for training on the dataset. The algorithm I am using for hyperparameter tuning, is ...
user avatar
2 votes
1 answer
50 views

Discrete, high dimensional optimization techniques

Suppose we have a binary vector of length 1000 where we can switch each element on or off. Each of the $2^{1000}$ different settings reveals a reward. The goal is to find the setting with the maximum ...
user avatar
  • 652
0 votes
1 answer
235 views

Question regarding log marginal likelihood in SKLearn

I'm trying to understand the hyperparameter optimization implemented in SKLearn. I'm using the basic example presented here with an alternative data set of 100 observations of Rastrigin test function (...
user avatar
  • 1
2 votes
1 answer
46 views

Is it valid to perform Bayesian optimization and MCMC sampling with nested model parameters?

I am trying to think of a way to perform a somewhat complicated Bayesian optimization problem involving model parameters which are nested within other functions in the forward problem, and I am ...
user avatar
  • 845
0 votes
1 answer
38 views

Bayesian Optimization with transformed objective function

I have a sequential learning problem where I want to rank a group of jobs in each iteration--unlike conventional Bayesian approach, I am trying to find the ordering of jobs based on g(f(x)) where GP ...
user avatar
  • 1
2 votes
2 answers
475 views

How does Bayesian Optimization balance exploration with exploitation?

I'm decently familiar with Gaussian Processes; I understand that GPs form the backbone of BO but I don't want this question to drift in scope towards an explanation of GPs. Rather, I'm curious, how ...
user avatar
  • 1,650
1 vote
1 answer
103 views

For variational autoencoder, how can we fix prior and likelihood distributions?

Before starting to describe my question, sorry for my poor English if you can't understand my descriptions. To my knowledge, generally, the Gaussian distribution is choosed as prior in VAE. Also, the ...
user avatar
  • 23
2 votes
0 answers
85 views

Why search techniques (besides Grid Search) are not generally used in ML?

From my experience and also by reading papers I noticed that in the Machine Learning / applied statistics world, when tuning hyperparameters there are basically two approaches: (1) some simple search ...
user avatar
2 votes
1 answer
1k views

Expected Improvement formula for Bayesian Optimisation

The expected improvement on how to choose a next point $x$ for evaluation is to choose the point such that $$arg\,max_{x}E[f(x) - f^{max}]$$ where $f(x)$ denotes the gaussian process posterior ...
user avatar
  • 876
4 votes
2 answers
665 views

Elementary explanation of Gaussian Processes

I'm learning of Gaussian Processes (GP) in the context of Bayesian Optimization (BO), where a surrogate function is learned to approximate the reward signal then a (hopefully) global optima is ...
user avatar
  • 1,650
0 votes
0 answers
175 views

Bayesian hyperparameter optimization, is it worth it?

On Goodfellow-et-al-2016,Deep Learning book the following sentence can be found: "Currently, we cannot unambiguously recommend Bayesian hyperparameter optimization as an established tool for ...
user avatar
4 votes
1 answer
190 views

Who first proposed Bayesian optimisation with Gaussian processes?

From what I understand, the 'standard' approach to Bayesian Optimisation uses a Gaussian process for the prior (as opposed to more recent proposals like TPE or Bayesian Optimisation with random ...
user avatar
  • 140
1 vote
0 answers
282 views

Binary classification, imbalanced dataset optimization: AUC vs logloss

I'm running optimization on an imbalanced dataset and need to define my optimization metric. I'm working on disease detection so maximizing AUC might not be the best solution, as the certainty of the ...
user avatar
  • 13
2 votes
0 answers
46 views

Maximising the utility when I have a large number of discrete decisions mixed with continuous decisions

I have an optimisation problem which is potentially quite tricky. Consider the problem of allocating a discrete number of resources (in my case, they are spare mechanical components for a large ...
user avatar
  • 2,275
0 votes
1 answer
68 views

Optimizing Equation involving CDF and PDF

Assume you have two random variables, $v_b \sim N(\mu_b,\sigma_b^2)$ and $v_s \sim N(\mu_s,\sigma_s^2)$ and let $\rho$ be the correlation between them. Denote by $F$ and $f$ the CDF and PDF of the ...
user avatar
0 votes
0 answers
49 views

Using Bayesian methods for experimental design

I'm trying to determine parameters for a function. After a given number of (computational) experiments the retrieved data is used for doing a fitting and the parameters are then obtained. (...
user avatar