Questions tagged [bayesian-optimization]

Bayesian optimization is a family of global optimization methods which use information about previously-computed values of the function to make inference about which function values are plausibly optima. Its applications include computer experiments and hyper-parameter optimization in some machine learning models.

Filter by
Sorted by
Tagged with
0
votes
0answers
11 views

Noob : Hyperparameter Bayesian Optimization for Stock-Trading Data? [closed]

My data consists of 5 Parameters ( each 3 - 5 values) => 500 unique Parameter-Combinations. I iterate those Parameter-Combinations over 250 different stocks and get Trading Results ( Profit). I get ...
0
votes
0answers
16 views

Expected Improvement for Sequential Model-based Global Optimization

I am reading a paper about hyper-parameter optimization (Bergstra et al., 2011), and I have a question about the expected improvement. In the paper, the pseudo-code of a generic SMBO is shown as ...
0
votes
0answers
17 views

Optimizing over a subset of input variables

Consider the unconstraint optimization problem: $$\arg \min_{𝑋_0} f(X_0, X_\eta)$$ where $X_0$ is the input vector I need to optimize for $𝑋_\eta$ is a nuisance vector which I do not care about. ...
1
vote
1answer
48 views

How to implement Bayesian Optimization when there is a relationship between search space parameters?

I'm quite new to Bayesian models. I'm trying to implement Bayesian Optimization where the sum of parameters should be less than a constant. Say, the parameters are ...
5
votes
2answers
166 views

Question of understanding regarding Bayesian Optimization, Gaussian process and acquisition function

I'm trying to understand Bayesian optimization and I struggle a lot with all the involved methods. Hence, I have some short questions: We start with a a-prior function, which is a gaussian process. A ...
0
votes
0answers
10 views

How to allocate the number of samples depending on the variances

I want to compute $p$ number of independent functions with each giving their result as expectation value, $e_{i=1,2..p}$. The final result is the sum of all expectation values, $S = \sum_{i}^{p}e_{i}$....
1
vote
0answers
50 views

Using a different (but related) hypothesis for the prior in MAP

Say we have a general set of data $\mathcal{D} = \{\mathbf{x}_i, \mathbf{y}_i \}_{i \in N}$ of covariates $\mathbf{x}$ and observations $\mathbf{y}$. Our problem is in fitting a known model $\mathbf{y}...
1
vote
0answers
27 views

Gaussian Process for noise free data

In a Gaussian Process model, the covariance function is given as follows: $$ C\left(y_{*} \mid y_{d}\right)=K\left(x_{*}, x_{*}\right)-K\left(x_{*}, x_{d}\right) K\left(x_{d}, x_{d}\right)^{-1} K\left(...
0
votes
0answers
9 views

Optimizing EI in the TPE algorithm

My question refers to the section "4.1 Optimizing EI in the TPE algorithm" on page 4 of the paper Algorithms for Hyperparameter Optimization (https://papers.nips.cc/paper/2011/file/...
1
vote
0answers
56 views

I am confused about the sum indices of these posterior distribution formulas

I am reading this notes and trying to understand how the posterior distribution formulas of the variational variables involved have been calculated. I am confused about the indices of the summation. ...
0
votes
0answers
22 views

What is the relationship between Bayesian Optimization and Sequential Model Based Optimization?

I often see these two terms being used almost interchangeably: Bayesian Optimization (BO) and Sequential Model Based Optimization (SMBO). Sometimes, SMBO is referred to as a formalisation/formalism of ...
0
votes
1answer
29 views

Bayesian optimization - One run with 100 iterations vs ten runs with ten iterations

I was wondering whether it makes more sense to run Bayesian Optimization let's say ones with 100 iterations or ten times with ten iterations each. Which one should I favor? I would assume that the ...
0
votes
0answers
26 views

why Gamma inverse is the conjugate prior of normal distribution?

I am trying to understand Bayesian regression. Then in Wikipedia enter link description here, it is written that by using the following relation we get the Gamma inverse as follow for the conjugate ...
0
votes
1answer
36 views

Which kinds of priors could be applied in Bayesian linear regression?

I would like to know can we use any kind of prior distribution in Bayesian linear regression and still convergent to a close mean solution to the Least square solution? Or it should be just a ...
3
votes
1answer
83 views

Bayesian Optimization vs. gradient descent

I don't know if this is the right place to ask this question. If you think this question is better asked in another StackExchange, please point me to that. This question is about the sampling ...
0
votes
0answers
12 views

Data acquisition for subsequent global/bayesian optimisation

For a research project, we are interested in applying global/bayesian optimisation for engineering an enzyme for optimal/maximum efficiency. The assumption is that there is a functional relationship $...
0
votes
0answers
11 views

Can't accurately estimate decay-fraction parameter from simulation using JAGS in R

I am generating simulated data of exponential decay of a variable over 10 sessions for 5 subjects, and seeing if I can estimate the 2 parameters (starting value and fraction of decay for each ...
1
vote
1answer
39 views

What distributions to use for hyperparameter tuning?

Where would one start when trying to figure out which distributions to use for hyperparameter tuning? Libraries such as HyperOpt, Optuna, and sklearn (random search) ask not for uniformly distributed ...
0
votes
1answer
19 views

Bayesian optmization when the relationship on some variables linear, and non-linear on other features

I am working on a Bayesian Optimization problem, where the approximator is GP, and the utility function is UCB1. My problem is the output of the function is linear on some features while non-linear on ...
1
vote
0answers
37 views

Can I use AdaBoost Regressor with Gaussian Process Regressor as base estimator, as my Surrogate Function?

I am using a text-extraction model whose hyperparamters I want to optimise. The model takes time(on average 1 hr) for training on the dataset. The algorithm I am using for hyperparameter tuning, is ...
1
vote
1answer
36 views

Discrete, high dimensional optimization techniques

Suppose we have a binary vector of length 1000 where we can switch each element on or off. Each of the $2^{1000}$ different settings reveals a reward. The goal is to find the setting with the maximum ...
0
votes
1answer
118 views

Question regarding log marginal likelihood in SKLearn

I'm trying to understand the hyperparameter optimization implemented in SKLearn. I'm using the basic example presented here with an alternative data set of 100 observations of Rastrigin test function (...
1
vote
1answer
39 views

Is it valid to perform Bayesian optimization and MCMC sampling with nested model parameters?

I am trying to think of a way to perform a somewhat complicated Bayesian optimization problem involving model parameters which are nested within other functions in the forward problem, and I am ...
0
votes
0answers
15 views

Is it a good idea to use the mean and standard deviation of coefficients from other models as my prior in Bayesian Regression?

I have a dataset that I’ve been playing around with for school I have gotten very good results with a bunch of methods (Ridge, Lasso, ElasticNet, SVM, Bagging, Stacking and NN even) Now I’m having a ...
0
votes
1answer
30 views

Bayesian Optimization with transformed objective function

I have a sequential learning problem where I want to rank a group of jobs in each iteration--unlike conventional Bayesian approach, I am trying to find the ordering of jobs based on g(f(x)) where GP ...
0
votes
0answers
13 views

Stochastic Gradient Langevin Dynamics Preconditioning Matrix time dependency

I am trying to implement the original Stochastic Gradient Langevin Dynamics paper (https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.441.3813&rep=rep1&type=pdf) to a problem I've ...
2
votes
2answers
168 views

How does Bayesian Optimization balance exploration with exploitation?

I'm decently familiar with Gaussian Processes; I understand that GPs form the backbone of BO but I don't want this question to drift in scope towards an explanation of GPs. Rather, I'm curious, how ...
0
votes
0answers
8 views

What parameters are initialized in Sparse Bayesian Learning?

I am reading few papers on Sparse Bayesian Learning topic. Many papers use posterior calculation or approximation using likelihood function and priors. Usually priors are defined like this P(w|Alpha) ...
1
vote
1answer
30 views

For variational autoencoder, how can we fix prior and likelihood distributions?

Before starting to describe my question, sorry for my poor English if you can't understand my descriptions. To my knowledge, generally, the Gaussian distribution is choosed as prior in VAE. Also, the ...
0
votes
0answers
15 views

Is it possible to use Bayesian Optimization directly on neural network parameters?

I'm using a deep neural network as a function approximator within a larger system which outputs a variable-size matrix. I have a black-box function, call it $J$, which gives me a real-valued cost for ...
2
votes
0answers
40 views

Why search techniques (besides Grid Search) are not generally used in ML?

From my experience and also by reading papers I noticed that in the Machine Learning / applied statistics world, when tuning hyperparameters there are basically two approaches: (1) some simple search ...
1
vote
1answer
418 views

Expected Improvement formula for Bayesian Optimisation

The expected improvement on how to choose a next point $x$ for evaluation is to choose the point such that $$arg\,max_{x}E[f(x) - f^{max}]$$ where $f(x)$ denotes the gaussian process posterior ...
4
votes
2answers
511 views

Elementary explanation of Gaussian Processes

I'm learning of Gaussian Processes (GP) in the context of Bayesian Optimization (BO), where a surrogate function is learned to approximate the reward signal then a (hopefully) global optima is ...
0
votes
0answers
85 views

Bayesian hyperparameter optimization, is it worth it?

On Goodfellow-et-al-2016,Deep Learning book the following sentence can be found: "Currently, we cannot unambiguously recommend Bayesian hyperparameter optimization as an established tool for ...
3
votes
1answer
107 views

Who first proposed Bayesian optimisation with Gaussian processes?

From what I understand, the 'standard' approach to Bayesian Optimisation uses a Gaussian process for the prior (as opposed to more recent proposals like TPE or Bayesian Optimisation with random ...
0
votes
0answers
79 views

Optimize learning rate of boosting algorithms efficiently with hyperopt

I've been exploring the hyperopt python library to tune parameters for boosting algorithms. Learning rate is one of the focal parameters for which efficient tuning appears more difficult. My ...
1
vote
0answers
133 views

Binary classification, imbalanced dataset optimization: AUC vs logloss

I'm running optimization on an imbalanced dataset and need to define my optimization metric. I'm working on disease detection so maximizing AUC might not be the best solution, as the certainty of the ...
2
votes
0answers
41 views

Maximising the utility when I have a large number of discrete decisions mixed with continuous decisions

I have an optimisation problem which is potentially quite tricky. Consider the problem of allocating a discrete number of resources (in my case, they are spare mechanical components for a large ...
0
votes
1answer
45 views

Optimizing Equation involving CDF and PDF

Assume you have two random variables, $v_b \sim N(\mu_b,\sigma_b^2)$ and $v_s \sim N(\mu_s,\sigma_s^2)$ and let $\rho$ be the correlation between them. Denote by $F$ and $f$ the CDF and PDF of the ...
0
votes
0answers
42 views

Using Bayesian methods for experimental design

I'm trying to determine parameters for a function. After a given number of (computational) experiments the retrieved data is used for doing a fitting and the parameters are then obtained. (...
1
vote
0answers
65 views

Black-box optimization algorithms performance

I've been doing some basic research of the black-box optimization algorithms that are on the market (and have Python implementation). I've found some recent and promising algorithms like LIPO, SHGO or ...
1
vote
0answers
24 views

Bayesian optimization experiment to confirm learning rate schedules in DNNs

Common learning rate schedules usually decrease the learning based on some criteria or some predefined schedule which intuitively makes sense. Has it been confirmed with bayesian hyper parameter ...
1
vote
1answer
194 views

Why not use large ($n - [n^{0.75}]$) validation sets in machine learning training loop?

I am seeking feedback in the form of answers to a bold question. Please limit comments to requests for clarification of the question only. There appears to be an influence on machine learning ...
0
votes
1answer
209 views

Bayesian optimization with xgb.cv and xgb.XGBClassifier - Mismatch between AUC scores

I'm doing bayesian hyperparameter optimization with bayes_opt and maximizing the AUC. I'm noticing a big discrepancy between the cross-validation scores that I ...
1
vote
0answers
76 views

Low rank plus diagonal approximation of covariance of model parameters

In this paper regarding Bayesian Deep Learning, in Section 3.4, the authors want to approximate the covariance of a distribution over the model parameters $\theta$. They first get an estimate of the ...
0
votes
2answers
84 views

Is Bayesian approach the correct way to solve this problem?

I have a dataset that has 4 variables and the target is a value between 1 to 5. this data is generated from an api that is black box and I don't know the formula that generates these values. I have ...
2
votes
1answer
121 views

Reduce search space for hyperopt

I am using the model SVR to create a regression model. This class contains several hyperparameters, and to try to find the best ones according to the several features, I am using the library hyperopt. ...
1
vote
1answer
28 views

Bayesian estimation difference between inferences is conjugate prior and non informative prior [duplicate]

What are the differences in the inferences using non-informative (specifically Jeffrys) priors as compared to the conjugate prior?
3
votes
2answers
35 views

Optimization of hyperparameters and parameters

As far as I understand, in Machine Learning there are 2 moments for optimization. Before training the model there is the optimization of the hyperparameters to find the best configuration of the model ...
1
vote
1answer
128 views

How to change priors in Bayesian estimation, if we get to know previous work was wrong?

Assuming we are doing a task to find how runs will be scored some match and for that, we have assumed some prior, now what if we find out midway through the process on which we built our priors was ...