Questions tagged [bayesian-optimization]

Bayesian optimization is a family of global optimization methods which use information about previously-computed values of the function to make inference about which function values are plausibly optima. Its applications include computer experiments and hyper-parameter optimization in some machine learning models.

Filter by
Sorted by
Tagged with
0
votes
1answer
55 views

why in Gaussian linear regression they put $y_i-\theta^Tx_i$ instead of $x$?

I am new to the machine learning area. Then I am studying a paper that considers the problem of logistic regression in Bayesian. In general, in regression or logistic regression when they assume the ...
1
vote
0answers
18 views

Expected improvement compared with probability of improvement and lower confidence bound

From all the references i read so far, i have been seeing Expected improvement being commonly used as acquisition function. Why is that so, compared with the other two i mentioned in the title/...
0
votes
0answers
11 views

Is sampling the training set for hyperparameter optimisation a good speed up solution?

I am using Bayesian hyperparameter optimization for LSTM hyperparameters. This manages to find an optimum set of hyperparameters in much fewer iterations than a grid search. My problem is that I ...
1
vote
2answers
68 views

What is the “tree” structure in Tree Parzen Estimators?

Context In Algorithms for Hyper-Parameter Optimization, the authors propose a "tree-structured" configuration space. Here, a configuration space is a space of hyperparameters. Questions What ...
1
vote
0answers
19 views

tradeoff function for exploration/exploitation in the context of gaussian process bayesian optimisation

First, let's give some context to my issue: consider an objective function $f(x)$ defined over a compact set $D$, such that there exists $x^*, f(x^*) > f(x) \quad \forall x \in D$. My goal is to ...
0
votes
0answers
6 views

Undersampling strategy as part of Bayesian optimization

I am using gp_minimize from skopt for the optimization of my model, which I then plug into ...
0
votes
1answer
102 views

Optimization problem with recursive functions

I could image that the following is a standard optimization problem but nevertheless I have no clue how to specifically solve it: by which specific approach, algorithm, and which computing powers I ...
2
votes
2answers
31 views

Optimizing hyperparameters of network with extremely long training time

As an example, let's say i am using a very deep fully convolutional autoencoder to segment lung scans. Input image resolutions will be large, since the features i hope to segment (things like early ...
2
votes
0answers
64 views

What is the current highest dimension for efficient Bayesian Optimization

It is frequently stated that BO can only be efficient for under 10 dimensions, or thereabouts. By efficient, I mean that the optimum is reached in an acceptable time, say less than a day given all ...
1
vote
1answer
60 views

acquisition function for bayesian optimisation using random forests as surrogate model

I'm working on implementing a Bayesian optimization class in Python. As a surrogate model, I used a Gaussian process until now. From what I read it's quite standard as it is efficient and intuitive. ...
0
votes
1answer
42 views

random forest regression with uncertainty output [closed]

For a Bayesian optimization problem, I wish to use a random forest regressor in Python, able to predict for unseen data a probability distribution, like in the paper Decision Forests for ...
1
vote
0answers
47 views

Gaussian Process - The variance term in Kernel matrix is not reducing to 0 at the Location of the Data Point

I am running a variance reduction experiment using GPy library and sampling using Bayesian optimization suggested location. I noticed that once the data at certain location is added to GP's data set, ...
0
votes
1answer
58 views

What is the actual distribution that we are modelling in case of a Bayesian Regression Model?

I have come across blog posts that speak about modelling a regression problem using Bayesian approaches. I completely understand that, to set up example data, they generate a sine wave using a one ...
0
votes
1answer
14 views

Using probabilistic scores in Bayesian Optimisation

I was reading up on Bayesian Optimization and in one of the articles, I came across the following passage. We could just use the surrogate score directly. Alternately, given that we have chosen a ...
-3
votes
2answers
93 views

Why are frequentists uncomfortable with bayesian statistics when “optimization” algorithms used in frequentist statistics is bayesian?

In Step 1, we have a prior. Using bayes rule we construct the posterior. In step 2 of some iterated bayesian procedure, the prior becomes the posterior from step one and use bayes rule to calculate ...
0
votes
0answers
24 views

Confusion between Acquisition function and Information Gain function in Bayesian Optimization

I am reading various papers that include some form of Bayesian information. So there are various acquisition function that are suitable for various applications such as follows: Expected Improvement ...
0
votes
0answers
24 views

Multi armed bandits with Thompson sampling and unknown rewards distribution

I have machine learning models that have a some possible post processing possibilities. I would like to use multi armed bandits to select the post processing that optimizes a continuous KPI. I have ...
1
vote
1answer
82 views

Multi armed bandits with natural ordering between bandits

Say I have a button in my website that I want to optimize the size given some metric. In order to use multi armed bandits for that, there are some things I would like to consider: My metric is a ...
0
votes
1answer
33 views

Is expected improvement acquisition function in Bayesian optimization a Gaussian process?

So I know there are several types of acquisition function for Bayesian optimization technique. But according to Wikipedia There are several methods used to define the prior/posterior distribution ...
3
votes
1answer
93 views

Difference between Bayesian Optimization and Bayesian Statistical Inference [closed]

Can someone give a lucid explanation pointing out how are Bayesian Optimization and Bayesian Statistical Inference different to each other? Can they be seen as complementary to each other or are they ...
2
votes
0answers
66 views

How can one optimize black-box functions given context?

Libraries like hyperopt or scikit-optimize allow one to optimize a black-box function. However, they do not allow specifying contextual information outside of the parameters to be chosen by the ...
0
votes
0answers
11 views

cross validation for hyperparameter selection: is it correct to split data to k-folds in each try differently?

I am doing hyperparameter selection with bayesian optimization. In each iteration I compute the cross validation error, which is objective to be minimized (or times -1 and will be maximized it does ...
0
votes
1answer
40 views

What do we want to maximize in hyperparameter tuning with bayesian optimization?

So I am trying to implement Bayesian optimization for various machine learning methods, all of then consist of hyperparameters which should be tuned (eg. complexity parameter, minimum samples in split ...
2
votes
1answer
104 views

How to estimate the error of the model in the best way?

I have many samples in my data and I want to fit many models which should be tuned by bayesian optimization. What is the best way for estimation of error, in statistical point of view? I mean, some ...
0
votes
0answers
14 views

Is it correct to choose different indices for cross validation in each iteration of bayesian optimization?

So I have implemented my first Bayesian Optimization for random forests in R. And now my question is whether it makes sense. I have used tune.randomForest function, which is normally for grid search,...
0
votes
1answer
141 views

Monte Carlo Sampling is performing better then Bayesian Optimization

So I am testing the Bayesian optimization library for determining where to sample next by quering a test function such as 2d Rosenbrock to better reconstruct that function using Gaussian Process ...
1
vote
0answers
54 views

Choosing a space function for hyperopt

I was originally doing a grid search for my parameter optimization and with 7 parameters being optimized, it would take ages. So I am choosing to use hyperopt at this point. I am however confused on ...
3
votes
1answer
38 views

Calculating the possible number of configurations

I am wondering how did they get the $19200$ possible configurations? Like, $5^6 = 15625$, where $6$ is the number of hyper-parameters: ps: just to check I'm doing the righ thing. Is this okay? The ...
1
vote
1answer
51 views

Bayesian update vs optimization

Say I have a multivariate normal vector $$ r \sim N(\mu , \Sigma ) \Rightarrow Pr \sim N(P\mu , P'\Sigma P ) $$ and I observe that $$ Pr = Q $$ Now I can use Bayes rule to calculate the ...
0
votes
1answer
153 views

Dirac Delta Function in Bayesian Optimization

I am reading a paper on Bayesian optimization which aims at selecting the batch of points $x_0,x_1,..,x_k$ to sample to obtain maxima or minima of the GP. So the first batch element is normal ...
0
votes
0answers
35 views

How does Bayesian provide Gaussian Process the capability of against overfitting

As frequently mentioned in books and papers, Gaussian Process for Regression has a nice built-in ability to against overfitting. This can be somewhat understandable solely from the equation ...
2
votes
1answer
53 views

What is Gaussian in BO: Can I use bayesian optimization for binary classification task?

BO works based on gaussian process, so what is gaussian there? Can I use BO for binary classification task, for example neural network which has output with logistic activation and is minimizing ...
1
vote
2answers
41 views

understanding bayesian optimization: what is meant by dimension

What is meant by the dimension in bayesian optimization? In many papers it is stated that the dimension should be lower than 20 (then there are papers which are solving high-dimensional problem). But, ...
0
votes
0answers
24 views

Is there an algorithm that can approximately search an 'isosurface' of a black box function $f$

Say I have a 2D(dimension can be higher) black box deterministic(no evaluation error) function f, I can evaluate every location's function value by f(x), though f do not have closed form expression. ...
0
votes
0answers
20 views

Bayesian Optimization Expression

I am reading a paper on Bayesian optimization and came across the following formula: Now my questions are: Question 1: Does the expression inside the redbox evaluate to $p(\mathcal I_{t,0})$? ...
0
votes
0answers
29 views

Marginal In Bayesian Optimization Expression

I am reading a paper and presentation on batch Bayesian optimization and came across the following formula. Question 1: Does the expression inside the redbox evaluate to $p(\mathcal I_{t,0})$? ...
0
votes
0answers
27 views

How to use estimated parameters obtained from Bayesian inference to make predictions

When using the Bayesian inference approach to identify the posterior distribution of the parameters, should we actually randomly sample each of the parameters from their corresponding posterior ...
1
vote
0answers
226 views

Bayesian hyperparameter optimization + cross-validation

I want to use Bayesian optimization to search a space of hyperparameters for a neural network model. My objective function for this optimization is validation set accuracy. In addition, I want to ...
0
votes
0answers
19 views

How to calculate or estimate RKHS norm?

I am working with GP-UCB and need to calculate RKHS norm as in Theorem 6 of Srinivas et.al 2012. I found on page 3 column 1 like: The induced RKHS norm $||{f}||_k=\sqrt{<f,f>}_k$ measures ...
1
vote
2answers
539 views

Intuitive Understanding of Expected Improvement for Gaussian Process

So I am learning Bayesian Optimization and came across expected improvement. My question is are we searching for the point in the Gaussian Process model whose expected value (determined by mean and ...
2
votes
2answers
76 views

Interpreting mixture of Gaussians (Variational Inference)

I've recently stated reading about mixture models and variational inference in this excellent paper, but I'm having troubles dissecting the models described, and have a couple of questions. Please see ...
2
votes
0answers
43 views

Gradient of the variance of a Gaussian process

I'm trying to compute the spatial derivatives of the expected improvement acquisition function in Gaussian-process optimisation, and doing so requires the spatial derivatives of the predictive ...
3
votes
3answers
191 views

Bayesian parameter estimation with proportion data

I am trying to do a Bayesian analysis using a model that comes from the literature in non-Bayesian form: $y = \Phi\Bigg(\frac{1}{\alpha} * log(A/\beta)\Bigg)$. Because the model uses the function $\...
0
votes
1answer
44 views

How to Derive SISO Kalman Filter Update Equation Using only Probability Density Functions

I'm trying to prove to myself that a single state/single measurement kalman update can be derived using bayes theorem (as proof of concept for a more complicated task) only using the PDF. I am able ...
0
votes
0answers
27 views

Multi-armed Bandit Algorithm selection and Optimization

I have 2 channels that I can sent my products, the A channel cost 0.10\$ per product and the B costs 0.01\$ and I am trying dynamically to optimize the channel selection by minimize the cost. ...
0
votes
0answers
9 views

Method to optimise portfolio WITHOUT mean or variance

Can anyone suggest a method to optimise a portfolio of stocks with high volatility. I would like to use the expected return, traded volume and delta from Mid point. Have looked at MPT and the would ...
6
votes
1answer
195 views

What is the relation between a surrogate function and an acquisition function?

A surrogate function is a simpler function than the objective function to evaluate. An acquisition function is used to propose sampling points. In the context of Bayesian optimisation and Gaussian ...
1
vote
0answers
235 views

How can I determine what values of alpha and kappa to use for Bayesian Optimization?

I'm using the pretty great Bayesian Optimization package for python. I have a very noisy function I'd like to optimize for a given hyperparameter. I've read a little on this, and it seems like if ...
0
votes
1answer
75 views

Bayesian optimization integer set constraint

Given an ordered input set of boolean values $S$ of length $|S|$, I want to minimize a function $f(S)$ while maximizing the number of $True \in S$ constrained by $|True \in S|$ $\ge$ $threshold$ ...
2
votes
1answer
24 views

Computing Sampling Cost Tradeoff

I'm working with a Bayesian Optimization problem where the accuracy of sampled data varies based on sampling cost (low-uncertainty data is expensive, high-uncertainty data is cheap). What kinds of ...