Questions tagged [bayesian-optimization]

Bayesian optimization is a family of global optimization methods which use information about previously-computed values of the function to make inference about which function values are plausibly optima. Its applications include computer experiments and hyper-parameter optimization in some machine learning models.

Filter by
Sorted by
Tagged with
0
votes
0answers
10 views

Is it a good idea to use the mean and standard deviation of coefficients from other models as my prior in Bayesian Regression?

I have a dataset that I’ve been playing around with for school I have gotten very good results with a bunch of methods (Ridge, Lasso, ElasticNet, SVM, Bagging, Stacking and NN even) Now I’m having a ...
0
votes
1answer
28 views

Bayesian Optimization with transformed objective function

I have a sequential learning problem where I want to rank a group of jobs in each iteration--unlike conventional Bayesian approach, I am trying to find the ordering of jobs based on g(f(x)) where GP ...
0
votes
0answers
5 views

Stochastic Gradient Langevin Dynamics Preconditioning Matrix time dependency

I am trying to implement the original Stochastic Gradient Langevin Dynamics paper (https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.441.3813&rep=rep1&type=pdf) to a problem I've ...
0
votes
1answer
35 views

How does Bayesian Optimization balance exploration with exploitation?

I'm decently familiar with Gaussian Processes; I understand that GPs form the backbone of BO but I don't want this question to drift in scope towards an explanation of GPs. Rather, I'm curious, how ...
0
votes
0answers
7 views

What parameters are initialized in Sparse Bayesian Learning?

I am reading few papers on Sparse Bayesian Learning topic. Many papers use posterior calculation or approximation using likelihood function and priors. Usually priors are defined like this P(w|Alpha) ...
0
votes
0answers
10 views

Multi Arm Bandits - Purchase Amount Zero Inflated

All the examples I have seen for multi-arm bandits (MAB) applied to say online advertising are for cases where the reward is binary (e.g. click, no click) or continuous (e.g. assuming a normal ...
1
vote
1answer
23 views

For variational autoencoder, how can we fix prior and likelihood distributions?

Before starting to describe my question, sorry for my poor English if you can't understand my descriptions. To my knowledge, generally, the Gaussian distribution is choosed as prior in VAE. Also, the ...
0
votes
0answers
14 views

Is it possible to use Bayesian Optimization directly on neural network parameters?

I'm using a deep neural network as a function approximator within a larger system which outputs a variable-size matrix. I have a black-box function, call it $J$, which gives me a real-valued cost for ...
2
votes
0answers
25 views

Why search techniques (besides Grid Search) are not generally used in ML?

From my experience and also by reading papers I noticed that in the Machine Learning / applied statistics world, when tuning hyperparameters there are basically two approaches: (1) some simple search ...
1
vote
1answer
118 views

Expected Improvement formula for Bayesian Optimisation

The expected improvement on how to choose a next point $x$ for evaluation is to choose the point such that $$arg\,max_{x}E[f(x) - f^{max}]$$ where $f(x)$ denotes the gaussian process posterior ...
4
votes
2answers
427 views

Elementary explanation of Gaussian Processes

I'm learning of Gaussian Processes (GP) in the context of Bayesian Optimization (BO), where a surrogate function is learned to approximate the reward signal then a (hopefully) global optima is ...
0
votes
0answers
34 views

Bayesian hyperparameter optimization, is it worth it?

On Goodfellow-et-al-2016,Deep Learning book the following sentence can be found: "Currently, we cannot unambiguously recommend Bayesian hyperparameter optimization as an established tool for ...
3
votes
1answer
51 views

Who first proposed Bayesian optimisation with Gaussian processes?

From what I understand, the 'standard' approach to Bayesian Optimisation uses a Gaussian process for the prior (as opposed to more recent proposals like TPE or Bayesian Optimisation with random ...
0
votes
0answers
31 views

Optimize learning rate of boosting algorithms efficiently with hyperopt

I've been exploring the hyperopt python library to tune parameters for boosting algorithms. Learning rate is one of the focal parameters for which efficient tuning appears more difficult. My ...
1
vote
0answers
38 views

Binary classification, imbalanced dataset optimization: AUC vs logloss

I'm running optimization on an imbalanced dataset and need to define my optimization metric. I'm working on disease detection so maximizing AUC might not be the best solution, as the certainty of the ...
2
votes
0answers
25 views

Maximising the utility when I have a large number of discrete decisions mixed with continuous decisions

I have an optimisation problem which is potentially quite tricky. Consider the problem of allocating a discrete number of resources (in my case, they are spare mechanical components for a large ...
0
votes
1answer
37 views

Optimizing Equation involving CDF and PDF

Assume you have two random variables, $v_b \sim N(\mu_b,\sigma_b^2)$ and $v_s \sim N(\mu_s,\sigma_s^2)$ and let $\rho$ be the correlation between them. Denote by $F$ and $f$ the CDF and PDF of the ...
0
votes
0answers
40 views

Using Bayesian methods for experimental design

I'm trying to determine parameters for a function. After a given number of (computational) experiments the retrieved data is used for doing a fitting and the parameters are then obtained. (...
1
vote
0answers
39 views

Black-box optimization algorithms performance

I've been doing some basic research of the black-box optimization algorithms that are on the market (and have Python implementation). I've found some recent and promising algorithms like LIPO, SHGO or ...
1
vote
0answers
20 views

Bayesian optimization experiment to confirm learning rate schedules in DNNs

Common learning rate schedules usually decrease the learning based on some criteria or some predefined schedule which intuitively makes sense. Has it been confirmed with bayesian hyper parameter ...
1
vote
1answer
131 views

Why not use large ($n - [n^{0.75}]$) validation sets in machine learning training loop?

I am seeking feedback in the form of answers to a bold question. Please limit comments to requests for clarification of the question only. There appears to be an influence on machine learning ...
0
votes
1answer
83 views

Bayesian optimization with xgb.cv and xgb.XGBClassifier - Mismatch between AUC scores

I'm doing bayesian hyperparameter optimization with bayes_opt and maximizing the AUC. I'm noticing a big discrepancy between the cross-validation scores that I ...
0
votes
0answers
21 views

Bayesian optimisation to find a function that satisfies some constraints

Am new to Bayesian optimisation and am trying to apply it to a research problem I am working on. General introductory literature to Bayesian optimisation suggests that we can use it to find the ...
0
votes
0answers
12 views

How to ensure inequality constraints on surrogate model from Bayesian optimization?

I am using bayesian optimization as a sequential strategy to globally optimize my objective function on a Simplex. Currently I am using Gaussian Process Regression for my surrogate model. Gaussian ...
1
vote
0answers
44 views

Low rank plus diagonal approximation of covariance of model parameters

In this paper regarding Bayesian Deep Learning, in Section 3.4, the authors want to approximate the covariance of a distribution over the model parameters $\theta$. They first get an estimate of the ...
0
votes
2answers
78 views

Is Bayesian approach the correct way to solve this problem?

I have a dataset that has 4 variables and the target is a value between 1 to 5. this data is generated from an api that is black box and I don't know the formula that generates these values. I have ...
2
votes
1answer
63 views

Reduce search space for hyperopt

I am using the model SVR to create a regression model. This class contains several hyperparameters, and to try to find the best ones according to the several features, I am using the library hyperopt. ...
0
votes
0answers
17 views

Experimental design for optimizing a machine producing candles [duplicate]

I have the following problem that I would like to solve, but I don't know where to look. I have a Machine A that produces candles. According to the pre-setting of the machine, the color, the height, ...
1
vote
1answer
23 views

Bayesian estimation difference between inferences is conjugate prior and non informative prior [duplicate]

What are the differences in the inferences using non-informative (specifically Jeffrys) priors as compared to the conjugate prior?
3
votes
2answers
29 views

Optimization of hyperparameters and parameters

As far as I understand, in Machine Learning there are 2 moments for optimization. Before training the model there is the optimization of the hyperparameters to find the best configuration of the model ...
0
votes
0answers
28 views

What is the most efficient algorithm for hyper-parameter optimization, specially when comparing Bayesian optimisation and evolutionary algorithms?

I've been reading some papers about how Bayesian optimisation has achieved great results in hyper-parameter tuning when compared with random search or grid search. What about when comparing with ...
0
votes
0answers
76 views

Understanding concrete dropout

I am trying to understand a technique call Concrete Dropout for automatically tuning the dropout rate during the network training. However, I am unable to follow the work described in the paper ...
0
votes
0answers
7 views

Bayesian estimation for analysing project evaluation

Using Bayesian estimation how can we assess the group projects given that different members in the group may have performed differently?
1
vote
1answer
76 views

How to change priors in Bayesian estimation, if we get to know previous work was wrong?

Assuming we are doing a task to find how runs will be scored some match and for that, we have assumed some prior, now what if we find out midway through the process on which we built our priors was ...
0
votes
1answer
42 views

Why does my Gibbs sampler find two optimals?

[EDITED] I am using a Gibbs Sampler to find a Bayesian optimization to my multilevel (hierarchical) model (2 levels). However, when I run multiple chains (each chain having different starting values) ...
2
votes
1answer
14 views

Bayesian optimization with offline stochastic function and sticky decisions

Is there a Bayesian optimization (BO) framework which allows: Warm start with offline data. The stochastic function $f(x)$ is noisy. Every iteration is $n$ samples controlled by agent and there are ...
0
votes
1answer
78 views

why in Gaussian linear regression they put $y_i-\theta^Tx_i$ instead of $x$?

I am new to the machine learning area. Then I am studying a paper that considers the problem of logistic regression in Bayesian. In general, in regression or logistic regression when they assume the ...
1
vote
0answers
52 views

Expected improvement compared with probability of improvement and lower confidence bound [closed]

From all the references i read so far, i have been seeing Expected improvement being commonly used as acquisition function. Why is that so, compared with the other two i mentioned in the title/...
0
votes
0answers
14 views

Is sampling the training set for hyperparameter optimisation a good speed up solution?

I am using Bayesian hyperparameter optimization for LSTM hyperparameters. This manages to find an optimum set of hyperparameters in much fewer iterations than a grid search. My problem is that I ...
4
votes
3answers
605 views

What is the “tree” structure in Tree Parzen Estimators?

Context In Algorithms for Hyper-Parameter Optimization, the authors propose a "tree-structured" configuration space. Here, a configuration space is a space of hyperparameters. Questions What ...
2
votes
0answers
52 views

tradeoff function for exploration/exploitation in the context of gaussian process bayesian optimisation

First, let's give some context to my issue: consider an objective function $f(x)$ defined over a compact set $D$, such that there exists $x^*, f(x^*) > f(x) \quad \forall x \in D$. My goal is to ...
0
votes
0answers
13 views

Undersampling strategy as part of Bayesian optimization

I am using gp_minimize from skopt for the optimization of my model, which I then plug into ...
0
votes
1answer
113 views

Optimization problem with recursive functions

I could image that the following is a standard optimization problem but nevertheless I have no clue how to specifically solve it: by which specific approach, algorithm, and which computing powers I ...
2
votes
2answers
70 views

Optimizing hyperparameters of network with extremely long training time

As an example, let's say i am using a very deep fully convolutional autoencoder to segment lung scans. Input image resolutions will be large, since the features i hope to segment (things like early ...
2
votes
0answers
67 views

What is the current highest dimension for efficient Bayesian Optimization

It is frequently stated that BO can only be efficient for under 10 dimensions, or thereabouts. By efficient, I mean that the optimum is reached in an acceptable time, say less than a day given all ...
1
vote
1answer
334 views

acquisition function for bayesian optimisation using random forests as surrogate model

I'm working on implementing a Bayesian optimization class in Python. As a surrogate model, I used a Gaussian process until now. From what I read it's quite standard as it is efficient and intuitive. ...
0
votes
1answer
267 views

random forest regression with uncertainty output [closed]

For a Bayesian optimization problem, I wish to use a random forest regressor in Python, able to predict for unseen data a probability distribution, like in the paper Decision Forests for ...
1
vote
0answers
88 views

Gaussian Process - The variance term in Kernel matrix is not reducing to 0 at the Location of the Data Point

I am running a variance reduction experiment using GPy library and sampling using Bayesian optimization suggested location. I noticed that once the data at certain location is added to GP's data set, ...
0
votes
1answer
59 views

What is the actual distribution that we are modelling in case of a Bayesian Regression Model?

I have come across blog posts that speak about modelling a regression problem using Bayesian approaches. I completely understand that, to set up example data, they generate a sine wave using a one ...
0
votes
1answer
17 views

Using probabilistic scores in Bayesian Optimisation

I was reading up on Bayesian Optimization and in one of the articles, I came across the following passage. We could just use the surrogate score directly. Alternately, given that we have chosen a ...