Linked Questions

11
votes
1answer
705 views

Which optimization algorithm to use for problems with many local optima and expensive goal function? [duplicate]

I've got an optimization problem with these properties: The goal function is not cheap to compute. It can be evaluated up to around 10^4 times in the optimization. There's a lot of local optima. High ...
1
vote
3answers
750 views

Neural Network for input values optimization [duplicate]

I have electric machine, which parameters I measure by 10 sensors. 8 of them measures "input" values and 2 of them result (output). I've got tons of historical data of all of these sensors. I built a ...
2
votes
1answer
220 views

Tuning of Hyperparameter [duplicate]

Are there any advanced packages that allows automated tuning of hyperparameters for neural network and traditional machine learning algorithms like XGBoost, random forest (using method like Bayesian, ...
0
votes
2answers
90 views

Optimization algorithm where calculating the function is very costly [duplicate]

I am doing physics simulation and have to minimize a function $f$ of approx. 10 parameters $f=f(x_1, x_2, ... , x_{10} )$. I have already looked into several optimization algorithms (downhill-simplex, ...
1
vote
2answers
51 views

Maximize slow function [duplicate]

So from experience (don't judge me, it was years ago...) I know that when a person with little knowledge about a field try to explain a problem to someone with a lot of experience it can easily lead ...
1
vote
0answers
72 views

Optimization without gradient [duplicate]

I have a non-convex function of several parameters, that takes real values. I am looking for a good algorithm to find local minima. I am reasonably confident I can come up with descent starting points....
1
vote
0answers
63 views

Is there any method for learning regularization parameter? [duplicate]

One of the hardest tasks in using machine learning methods is choosing the appropriate hyper-parameters of the model such as regularization parameter. As far as I know, this task is performed by a ...
0
votes
0answers
16 views

Deriving cost function [duplicate]

I have calculated data from a computer simulation for a wireless network as shown below, here P is an algorithm tuning parameter, 'C' and 'D' are % decrease in collisions and % decrease in delay ...
81
votes
11answers
9k views

Why should I be Bayesian when my model is wrong?

Edits: I have added a simple example: inference of the mean of the $X_i$. I have also slightly clarified why the credible intervals not matching confidence intervals is bad. I, a fairly devout ...
51
votes
6answers
26k views

Practical hyperparameter optimization: Random vs. grid search

I'm currently going through Bengio's and Bergsta's Random Search for Hyper-Parameter Optimization [1] where the authors claim random search is more efficient than grid search in achieving ...
24
votes
4answers
9k views

When are genetic algorithms a good choice for optimization?

Genetic algorithms are one form of optimization method. Often stochastic gradient descent and its derivatives are the best choice for function optimization, but genetic algorithms are still sometimes ...
23
votes
2answers
12k views

How could stochastic gradient descent save time compared to standard gradient descent?

Standard Gradient Descent would compute gradient for the entire training dataset. ...
15
votes
3answers
7k views

How to run linear regression in a parallel/distributed way for big data setting?

I am working on a very large linear regression problem, with data size so large that they have to be stored on a cluster of machines. It will be way too big to aggregate all the samples into one ...
11
votes
3answers
552 views

Optimization of stochastic computer models

This is a tough topic for myself to google since having the words optimization and stochastic in a search almost automatically defaults to searches for stochastic optimization. But what I really want ...
11
votes
1answer
3k views

What are some of the disavantage of bayesian hyper parameter optimization?

I am fairly new to machine learning and statistics but I was wondering why bayesian optimization is not referred more often online when learning machine learning to optimize your algorithm ...
1
vote
4answers
15k views

SVM Hyperparameters Tuning

I am using SVM classifier to classify data, My dataset consist of about 1 milion samples, Currently im in the stage of tunning the machine , Try to find the best parameters including a suitable ...
5
votes
1answer
9k views

hyperparameter tuning in neural networks

I was trying to fine tune a neural network model for a multilabel classification problem. I was reading Jason Brownlee 's article for the same. As per the article, there are a number of parameters to ...
11
votes
2answers
5k views

Markov chain Monte Carlo (MCMC) for Maximum Likelihood Estimation (MLE)

I am reading a 1991 conference paper by Geyer which is linked below. In it he seems to elude to a method that can use MCMC for MLE parameter estimation This excites me since, I have coded BFGS ...
10
votes
3answers
347 views

Best way to evaluate PDF estimation methods

I wish to test some of my ideas that I think are better than anything that I have seen. I could be wrong but I'd like to test my ideas and vanquish my doubts by more certain observations. What I have ...
3
votes
1answer
6k views

Using genetic algorithm for hyperparameter optimization

In machine learning, I've learned one of the ways to optimize hyperparameters of a model is to do a grid search, which tests model for evenly spaced out values of hyperparametrs and determines which ...
5
votes
1answer
2k views

Optimizing a “black box” function: Linear Regression or Bayesian Optimization… what's the difference?

Goal: I have a function $f(x,y)=z$ (two variables for illustration only) which I know almost nothing about--it has a compact domain which I can determine, it is non-negative, and bounded above. My ...
2
votes
1answer
1k views

hyper parameter optimization grid search issues

I keep running into the same problem while doing a grid search to optimize the C and gamma parameters of an SVC. Every time i do the grid search, the best values seem to occur at around C = 100000 ...
2
votes
2answers
983 views

Faster xgb.cv for large data set

I have a record of data that contains 1.1 millions observations and 14 variables. The response is 0 or 1. It was suggested to me that I use Gradient Boosted Trees to build my logistic model. Using <...
3
votes
1answer
753 views

Bayesian Hyperparameter Tuning

I would like to implement Bayesian hyperparameter tuning in Python, but I am struggling to find a complete explanation of how it is exactly working. As anyone links, books or papers I could read about ...
2
votes
1answer
1k views

How to optimize RBF parameters $C,\gamma$ with KSVM method?

I want to find the best choice of $C$ and $\gamma$ parameters for Radial Basis Function kernel. I am using kernlab instead of e1071 library. So how can i optimize RBF parameters $C$ and $\gamma$ with ...
1
vote
1answer
468 views

Is the AUC a robust metric to determine which network architecture to use?

I would like to figure out which hyperparameters/network architectures work best on a specific binary dataset. To compare them, I do k-fold cross-validation, use different MLP or ConvNet ...
5
votes
2answers
630 views

Regression: find the best degree of polynomial with the best regularization parameter

When trying to predict data using linear regression or classify with logistic regression, with a polynomial, I know how to find the best degree of a polynomial to fits given data when the ...
1
vote
2answers
611 views

Maximization of Output based on Input

What I want to do is find the values for $X = $ { $x_j$ } that will produce the maximum $y$. I'm currently trying to maximize my output $y$, based on my inputs $X$. Say there are inputs, $X = $ { $...
3
votes
2answers
602 views

sensitivity analysis on given data [closed]

Is there any efficient method to do global parameter sensitivity analysis based on given data, without generating new cases to simulate (very expensive in my case). Indeed, a lot of famous ...
1
vote
1answer
443 views

2018 Update - Different Algorithms for Hyperparameter Optimization

Common question: What are the different options (in common languages like R or Python) available for optimizing hyperparameters? I am primarily interested in implementations in R that can work with ...
4
votes
1answer
160 views

What optimization (maximization/minimization) methods exist contours with lots of kinks?

I am wondering what methods exist out there for optimization on problems where your contour has lots of small sharp kinks. For example, suppose we have a contour where there are lots of lots of spiky, ...
0
votes
1answer
682 views

How do we use cross-validation to get hyper parameters for soft-margin Gaussian kernel SVM?

I was reading this How to select kernel for SVM? but as I am relatively new to the applied techniques in ML (only familiar with some theory) I may need some explaining on the answers. Specifically, ...
2
votes
0answers
735 views

Maximum Prediction in Gaussian Process

A Gaussian process (GP) is defined as a collection of random variables with a joint Gaussian distribution (Rasmussen 2006). It is well known that given observations $\left \{ \mathbf{x},\mathbf{y}\...
2
votes
2answers
317 views

Best DoE method to fit Gaussian Process Regressor

I would like to fit a Gaussian Process Regressor from sk-learn to predict values of an the objective function in order to explore behaviour and interplay between inputs. Data points are difficult to ...
2
votes
1answer
294 views

Optimizing black box functions without Bayesian Optimization

I have a black box system which accepts a data instance $\boldsymbol{x}$ and outputs a corresponding $y$. I don not know the exact mathematical formula of the system. What I want to do is maximizing $...
0
votes
1answer
259 views

best practices for baysian optimization of hyper parameters DNN [closed]

Many people suggest fine-tuning a network using Bayesian optimization ( or grid search or what every other black box optimization method you like ) so I tried it for my self. I am not sure about the ...
1
vote
1answer
217 views

Is it posible to obtain the next point without bruteforcing in Bayesian Optimization?

The implementations I have found till now of Bayesian optimization tipically find the optimal input point X for next step, which minimizes the output Y of the acquisition function which is being ...
-1
votes
1answer
212 views

Improvements of Random Search for Hyperparameter Optimization [closed]

Random search is one possibility for hyperparameter optimization in machine learning. I have applied random search to search for the best hyperparameters of a SVM classifier with a RBF kernel. ...
0
votes
2answers
97 views

MLE over a large number of parameters

I am working on a model that I am fitting to a medium-sized dataset ($\approx 300,000$ observations). The likelihood I have derived is a mixture model of some Beta distributions which also has the ...
1
vote
1answer
203 views

Are there are alternatives to gradient update rule?

Most optimization techniques (that I'm aware of) for non-linear cost functions that are commonly implemented rely on linearly updating a variable iteratively until a minimum is reached or a condition ...
3
votes
0answers
228 views

Hyperparameter optimization in 6-dimensional continuous space

I am using Random Forest and Stochastic Gradient Boosting to predict a categorical target variable exhibiting severe between-class imbalance. I am using oversampling to make sure that the models do ...
2
votes
0answers
178 views

Multi-start optimization

I am performing $k$-fold cross-validation for model selection via maximum-likelihood estimation. At the moment I am using a standard optimizer (fmincon in MATLAB). Since the likelihood landscape is ...
1
vote
2answers
50 views

Algorithm to determine program's stable surface

Assume I have a program which we'll model as bool Worked = f(param1,param2,...) Assume these parameters are positive integers, and a starting set exists s/t Worked ...
2
votes
1answer
78 views

Finding the maximum of a function $f(x)$ without analytically evaluating $f'(x)$

I'm an experimental physicist, trying to automate a relatively simple (but sensitive) optimization in my measurements that is currently done completely manually and takes up a lot of my time. I figure ...
3
votes
1answer
74 views

What is Gaussian in BO: Can I use bayesian optimization for binary classification task?

BO works based on gaussian process, so what is gaussian there? Can I use BO for binary classification task, for example neural network which has output with logistic activation and is minimizing ...
1
vote
0answers
85 views

Using a surrogate model for the solution space of an optimization problem

I have an optimization problem: Given a complex $n\times n$ covariance matrix $C$ one must find a complex $n$-vector $v_C^\ast$ which (approximately) minimizes an objective $f_C(v)$ over all space. $...
1
vote
0answers
82 views

Confidence intervals and central estimates for a functional of an estimated function with uncertain parameters

I've got a problem that is leading me to dip my toes into Bayesian stats, and I've got a question about confidence (or, I suppose, credible) intervals: Say you want to know how $X$ maps to $y$. You ...
0
votes
0answers
74 views

Machine learning algorithm to replicate a CPU-intensive, deterministic set of calculations

I am working on a data visualization like this NY Times interactive (http://www.nytimes.com/interactive/2014/upshot/buy-rent-calculator.html): If you go to that link, there are 20 of these charts. I ...
1
vote
1answer
53 views

Least squares optimization with expensive model and many parameters

I have a physical model which takes $\sim50$ parameters and gives $\sim 2000$ outputs taking tens of minutes to run. I need to optimise these parameters to give outputs as close as possible to data by ...
1
vote
2answers
49 views

What kind of optimization am I using?

I would like to know under optimization taxonomy, where do the following method fall? I have a combinatarial optimization problem, that I am trying to solve, Maximize an objective value for the ...

15 30 50 per page