Questions tagged [bayesian-optimization]

Bayesian optimization is a family of global optimization methods which use information about previously-computed values of the function to make inference about which function values are plausibly optima. Its applications include computer experiments and hyper-parameter optimization in some machine learning models.

Filter by
Sorted by
Tagged with
0 votes
0 answers
15 views

Periodic oscillations or discontinuities in likelihood function / loglikelihood function sampling over 2D grid of data

Is it expected behaviour that the loglikelihood function may have periodic oscillations within an envelope, or even periodic discontinuities for data sampled over a 2D grid? For example, generate mock ...
  • 1
1 vote
1 answer
97 views

Minimum sampling for maximising the prediction accuracy [closed]

Suppose that I'm training a machine learning model to predict people's age by a picture of their faces. Lets say that I have a dataset of people from 1 year olds to 100 year olds. But I want to choose ...
  • 61
0 votes
0 answers
18 views

How to create a detection probability matrix for hypothesis testing

In Convex Optimization by Boyd there is a section on optimal detectors and hypothesis testing. I am having a hard time understanding how the randomized detectors work given the formulas. The section ...
  • 101
1 vote
0 answers
23 views

How to solve this type of multi-task Bayesian optimization problem?

Let us consider a collection of local Bayesian optimization tasks, each employs a Gaussian Process model to find the local optimum (i.e. global optimum of that task). The goal is to design a ...
  • 163
0 votes
0 answers
12 views

How to evaluate likelihood in MCMC for arbitrarily shaped distributions?

I'm very confused with the use of MCMC to estimate distributions that have a complex shape, like multiple peaks, or that aren't generated from a known distribution. In particular, when calculating the ...
1 vote
0 answers
32 views

What is global concavity of the (log-)likelihood worth in Bayesian estimation?

In maximum likelihood estimation there is a big emphasis on finding the global maximum, which is why likelihood functions that are provably globally (log-)concave are desirable (despite often being ...
  • 1,000
0 votes
0 answers
126 views

which dataset to send as eval set in xgboost, catboost, and etc, when using optuna

In some boost models there are option to send eval set while fitting the model. for example: ...
  • 77
2 votes
1 answer
34 views

Is it correct to replace the oldest data when using Gaussian Process in blackbox hyper-parameter optimization?

I try to apply GPR in a blackbox HPO question. My input will have 6 dimensions like X=[x1,...x6]. The implementation is quite straightforward with sklearn with a ...
  • 145
0 votes
0 answers
32 views

How to restore Optuna's finished study from logs?

I wish to restore optuna.study.Study class object from uncomplete (5000 last lines, which is enough for my goals) logs. They look like that: ...
0 votes
0 answers
15 views

Using a kenel function which is different from the common set of kernels

The function $f:\mathbb{R}\rightarrow\mathbb{R}$ is a neural network that depends on $\theta$ (model parameters) and $\lambda$ (hyperparameters). Therefore, the surrogate function can be written as ...
  • 101
1 vote
1 answer
165 views

How to run Bayesian optimization experiments in parallel?

Suppose I have the following hyperparameters for tuning: learning_rate: [0.00001, 0.1] epochs: [200,300,400,....,1000] batch_size: [16,32,64,128] If I want to run experiments using 4 parallel jobs, ...
  • 867
1 vote
0 answers
23 views

How much reduction of hyperparameter experiments can I get using Bayesian optimization vs Grid search?

I have 5 hyperparameters for tuning and the number of combinations of all possible values is 9,360. This means if I want to find the optimal parameter setting using Grid Search, I need to do 9,360 ...
  • 867
0 votes
0 answers
44 views

Why is it desirable to standardize the inputs of a Gaussian Process Regression?

I read this question (Should we standardize the data while doing Gaussian process regression?) and wondered, why do we need to normalize the inputs of a Gaussian Process? In my case, I want to use ...
  • 1
0 votes
0 answers
72 views

Implementation of bayesian optimization

Is there an easy to apply implemented tool in python or R for a bayesian optimization? As I know this topic only superficially, I want to use bayesian optimization to determine the number of ...
  • 2,441
2 votes
1 answer
22 views

Clarification on Bayesian ensembling

I was reading a paper https://arxiv.org/pdf/2007.06823.pdf and at the end of page 3 the author presents the technique called "ensembling" for the estimation of the expected outputs and the ...
  • 207
0 votes
0 answers
62 views

what results are still usable from non-nested cross validation when tuning hyperparameters (and reconciling that with Optuna)

I'm trying to wrap my head around nested cross validation for the purposes of hyperparameter tuning. Part i) If I was to run hyperparameter tuning with cross validation (without nesting), say with ...
  • 111
1 vote
1 answer
94 views

Is it possible to pick more than 1 sample point in each iteration of bayesian optimisation?

I want to use Bayesian optimisation for my project and I plan to build a closed-loop system, such that there is a model, robot to conduct experiments, measurement of experimental data which updates ...
  • 31
3 votes
1 answer
82 views

How does removal of symmetry (e.g. via constraints) in a Bayesian optimization search space affect search efficiency?

There are many examples of search space symmetry in real-world optimization problems in the physical sciences. To motivate this, here are some that come to mind: When optimizing a formulation such as ...
1 vote
1 answer
120 views

Package for hyperparameter optimization with categorical values

Context I'm trying to solve a black-box optimization problem, and I can "reformulate" parts of the problem is different ways that may lead to lower or higher costs, and which can interact ...
1 vote
0 answers
30 views

Use of Monte Carlo Tree Search

I was talking with someone much more experienced in stats than I am and they suggested the use of Monte Carlo Tree Search for a problem I am facing. Problem Statement: I am collecting jitter ...
0 votes
0 answers
36 views

Fixed-leg Kalman filter smoother (Rauch–Tung–Striebel) error bounds

Although very intuitive and with plenty of results that talk about the asymptotic convergence of the estimate I wasn't able to track down any paper stating explicitly convergence bounds based on the ...
  • 1
0 votes
0 answers
18 views

Discussion of hyperparamter to optimize

For my thesis I consider to add a small discussion about hyperparameter optimization (HPO). I have five tree based models I would like to tune. Namely RandomForest, GBM, XGBoost, LightGBM and CatBoost....
  • 218
1 vote
1 answer
42 views

What is the difference between Bayesian Optimizaiton for hyperparameters and using validation set when training?

While studying hyperparameter tuning in Machine Learning, I have come to read Bayesian Optimization for Hyperparameter Tuning and Using validation set when training the model but it is kind of ...
  • 2,655
0 votes
0 answers
151 views

Bayesian optimization resulting in overfitting despite split test train

I am using XGBRegressor on a time series to predict the next periods value. I use sklearns TimeSeriesSplit to split the data into test and train. I use skopts BayesSearchCV to search through a ...
0 votes
2 answers
80 views

Is the result of Bayesian Inference with MCMC reliable since there maybe a big variance?

I'm new to Bayesian inference and I came across a, maybe simple, question. That is, in many cases, we don't know how the posterior distribution really looks like, and we adapt the result of MCMC, or ...
  • 101
0 votes
0 answers
34 views

How to solve optimization problem?

I am trying to implement the following from a paper I am reading. I have reached the following part, but I am unsure of how to solve the optimization problem. The authors do not specify what method ...
1 vote
1 answer
101 views

Gaussian process on hyperparameter tuning

I feel it is kind of circular to use GP for hyperparameter tuning, since GP has its own hyperparameters. Or is it the case that GP typically has less number of hyperparameters than the model we want ...
  • 223
0 votes
0 answers
27 views

Why Bayesian optimization in its simplest form only works for low dimensional optimization? [duplicate]

This is what I learned from the class on how posterior is updated for Gaussian process: It seems the computational cost of the matrix inversion only scales with the size of K_t, which depends on the ...
  • 223
0 votes
0 answers
67 views

Expected Improvement for Sequential Model-based Global Optimization

I am reading a paper about hyper-parameter optimization (Bergstra et al., 2011), and I have a question about the expected improvement. In the paper, the pseudo-code of a generic SMBO is shown as ...
0 votes
0 answers
33 views

Optimizing over a subset of input variables

Consider the unconstraint optimization problem: $$\arg \min_{𝑋_0} f(X_0, X_\eta)$$ where $X_0$ is the input vector I need to optimize for $𝑋_\eta$ is a nuisance vector which I do not care about. ...
  • 101
1 vote
1 answer
60 views

How to implement Bayesian Optimization when there is a relationship between search space parameters?

I'm quite new to Bayesian models. I'm trying to implement Bayesian Optimization where the sum of parameters should be less than a constant. Say, the parameters are ...
7 votes
2 answers
418 views

Question of understanding regarding Bayesian Optimization, Gaussian process and acquisition function

I'm trying to understand Bayesian optimization and I struggle a lot with all the involved methods. Hence, I have some short questions: We start with a a-prior function, which is a gaussian process. A ...
  • 2,441
1 vote
0 answers
52 views

Using a different (but related) hypothesis for the prior in MAP

Say we have a general set of data $\mathcal{D} = \{\mathbf{x}_i, \mathbf{y}_i \}_{i \in N}$ of covariates $\mathbf{x}$ and observations $\mathbf{y}$. Our problem is in fitting a known model $\mathbf{y}...
1 vote
0 answers
45 views

Gaussian Process for noise free data

In a Gaussian Process model, the covariance function is given as follows: $$ C\left(y_{*} \mid y_{d}\right)=K\left(x_{*}, x_{*}\right)-K\left(x_{*}, x_{d}\right) K\left(x_{d}, x_{d}\right)^{-1} K\left(...
0 votes
1 answer
90 views

Optimizing EI in the TPE algorithm

My question refers to the section "4.1 Optimizing EI in the TPE algorithm" on page 4 of the paper Algorithms for Hyperparameter Optimization (https://papers.nips.cc/paper/2011/file/...
  • 161
1 vote
0 answers
59 views

I am confused about the sum indices of these posterior distribution formulas

I am reading this notes and trying to understand how the posterior distribution formulas of the variational variables involved have been calculated. I am confused about the indices of the summation. ...
1 vote
1 answer
131 views

Bayesian optimization - One run with 100 iterations vs ten runs with ten iterations

I was wondering whether it makes more sense to run Bayesian Optimization let's say ones with 100 iterations or ten times with ten iterations each. Which one should I favor? I would assume that the ...
  • 533
0 votes
0 answers
51 views

why Gamma inverse is the conjugate prior of normal distribution?

I am trying to understand Bayesian regression. Then in Wikipedia enter link description here, it is written that by using the following relation we get the Gamma inverse as follow for the conjugate ...
  • 143
0 votes
1 answer
63 views

Which kinds of priors could be applied in Bayesian linear regression?

I would like to know can we use any kind of prior distribution in Bayesian linear regression and still convergent to a close mean solution to the Least square solution? Or it should be just a ...
  • 143
3 votes
1 answer
442 views

Bayesian Optimization vs. gradient descent

I don't know if this is the right place to ask this question. If you think this question is better asked in another StackExchange, please point me to that. This question is about the sampling ...
  • 161
1 vote
1 answer
214 views

What distributions to use for hyperparameter tuning?

Where would one start when trying to figure out which distributions to use for hyperparameter tuning? Libraries such as HyperOpt, Optuna, and sklearn (random search) ask not for uniformly distributed ...
  • 113
1 vote
1 answer
48 views

Bayesian optmization when the relationship on some variables linear, and non-linear on other features

I am working on a Bayesian Optimization problem, where the approximator is GP, and the utility function is UCB1. My problem is the output of the function is linear on some features while non-linear on ...
1 vote
0 answers
69 views

Can I use AdaBoost Regressor with Gaussian Process Regressor as base estimator, as my Surrogate Function?

I am using a text-extraction model whose hyperparamters I want to optimise. The model takes time(on average 1 hr) for training on the dataset. The algorithm I am using for hyperparameter tuning, is ...
2 votes
1 answer
57 views

Discrete, high dimensional optimization techniques

Suppose we have a binary vector of length 1000 where we can switch each element on or off. Each of the $2^{1000}$ different settings reveals a reward. The goal is to find the setting with the maximum ...
  • 651
0 votes
1 answer
395 views

Question regarding log marginal likelihood in SKLearn

I'm trying to understand the hyperparameter optimization implemented in SKLearn. I'm using the basic example presented here with an alternative data set of 100 observations of Rastrigin test function (...
  • 1
2 votes
1 answer
73 views

Is it valid to perform Bayesian optimization and MCMC sampling with nested model parameters?

I am trying to think of a way to perform a somewhat complicated Bayesian optimization problem involving model parameters which are nested within other functions in the forward problem, and I am ...
  • 845
1 vote
1 answer
46 views

Bayesian Optimization with transformed objective function

I have a sequential learning problem where I want to rank a group of jobs in each iteration--unlike conventional Bayesian approach, I am trying to find the ordering of jobs based on g(f(x)) where GP ...
  • 11
2 votes
2 answers
758 views

How does Bayesian Optimization balance exploration with exploitation?

I'm decently familiar with Gaussian Processes; I understand that GPs form the backbone of BO but I don't want this question to drift in scope towards an explanation of GPs. Rather, I'm curious, how ...
  • 1,728
1 vote
1 answer
166 views

For variational autoencoder, how can we fix prior and likelihood distributions?

Before starting to describe my question, sorry for my poor English if you can't understand my descriptions. To my knowledge, generally, the Gaussian distribution is choosed as prior in VAE. Also, the ...
  • 23
2 votes
0 answers
123 views

Why search techniques (besides Grid Search) are not generally used in ML?

From my experience and also by reading papers I noticed that in the Machine Learning / applied statistics world, when tuning hyperparameters there are basically two approaches: (1) some simple search ...