Questions tagged [optimization]

Use this tag for any use of optimization within statistics.

Filter by
Sorted by
Tagged with
33 votes
4 answers
10k views

Why does Bayesian Optimization perform poorly in more than 20 Dimensions?

I have been studying Bayesian Optimization lately and made the following notes about this topic: Unlike deterministic functions, real world functions are constructed using physical measurements ...
12 votes
2 answers
17k views

How to set limits using constrOptim in R?

I am using constrOptim to minimize a log likelihood function for maximum likelihood estimation of parameters. I wish to set the bounds on my parameters, but to not understand the constrOptim ...
0 votes
0 answers
36 views

multiple testing and p-hacking when choosing from many models

I'm trying to train a random forest model with a particle swarm optimization algorithm because my target function is not smooth and unknown to me, that is, in essence, this is search and training at ...
2 votes
2 answers
587 views

permutation testing a nested cross validation SVM

Say that I used a nested cross validation to do SVM classification on an fMRI dataset with hyperparameter tuning ( using a linear or rbf kernel). The classification on my outer cross validation folds ...
5 votes
1 answer
1k views

How does Canonical Time Warping help in time alignment?

Canonical Time Warping is a state-of-the-art technique for time alignment. According to the original paper, it helps account for individual varieties when aligning sequences derived from different ...
3 votes
1 answer
592 views

Bayesian Regression- Expectation Maximization

In Bayesian regression, we have $y_i=x_i^{T}w+\epsilon_i$ where $w \sim \mathcal{N}(0,\alpha)$ and $\epsilon_i \sim \mathcal{N}(0,\frac{1}{\beta})$. Inference of $\alpha$ and $\beta$ is done by ...
252 votes
10 answers
127k views

Why is Newton's method not widely used in machine learning?

This is something that has been bugging me for a while, and I couldn't find any satisfactory answers online, so here goes: After reviewing a set of lectures on convex optimization, Newton's method ...
0 votes
0 answers
11 views

Is there room for finding a more efficient hybrid optimization problem, in the context of optimization algorithms for MLE?

Recently finished my statistical modelling class, but it only briefly touched on Maximum Likelihood Estimates and I thought it was an interesting topic, so I decided to go deeper in my own time. I ...
0 votes
0 answers
12 views

Expectation over cost-normalized Expected improvements

Are the following two expressions equivalent if we assume the independence of f(x) and C(x)? $$ E\left[\frac{E\left[\max\left(f(x) - f(x^*), 0\right)\right]} {C(x)}\right] $$ $$ \frac{E\left[\max\...
2 votes
0 answers
32 views

Posterior approximation following optimization methods

I'm trying to quantify the uncertainty in a high dimensional, and multimodal posterior space. We do not have a analytical solution for the forward model, and the forward model could be expensive to ...
6 votes
1 answer
2k views

Momentum vs Polyak averaging

I'm going through this deck but don't quite get the difference between momentum and Polyak averaging, and what role Polyak averaging plays in modern optimizers. For example, is it correct to say that ...
0 votes
0 answers
16 views

Error term in SGD with momentum

I am reading the article "How Momentum really works" (https://distill.pub/2017/momentum/), and i am confused in one point: I am trying to derive the convergence rate for momentum from the ...
2 votes
1 answer
245 views

What optimization algorithms are best at traversing complicated geometries, and what trade-offs exist between different algorithms?

Hi all :] so short version is just the title, and specifically as pertains to those algorithms included in R's optimx::optimx() under ...
2 votes
1 answer
524 views

Why do increasing regularization weights make objective function not monotonically decrease?

I run modified non-negative matrix factorization (NMF) and tune the regularization weight from 1e5 to 1e13. The table below ...
5 votes
1 answer
2k views

Confusion about Robbins-Monro algorithm in Bishop PRML

This is basically how Robbins-Monro is presented in chapter 2.3 of Bishop's PRML book (from his slides): In the general update equation, $$ \theta^{(N)} = \theta^{(N-1)} - \alpha_{N-1}z(θ^{(N-1)}) $$ ...
3 votes
1 answer
510 views

Proof that log-odds minimize binomial deviance

How do you prove that minimizing the binomial deviance estimates the log-odds? i.e: $$ \ln{\left ( \frac{p(x_i)}{1-p(x_i)} \right )} = \underset{f(x_i)}{argmin} \ \mathbb{E} \left [y_i \ln \left ( \...
1 vote
0 answers
14 views

Bayesian Optimization using randomForest surrogate model in R language is taking a very longer time to complete [closed]

I am running a Bayesian Optimization to optimize an objective function where the difference between the predicted validation set and the mean of initial output of the dataset is kept to the bearest ...
5 votes
1 answer
265 views

Examples in the Real World where Evolutionary Algorithms/Genetic Algorithms Outperform other Classes of Optimization Algorithms

I have been trying to do some research to find out if there are certain industries/types of problems or even specific examples in applied research paper where Evolutionary Algorithms (e.g. Genetic ...
0 votes
0 answers
39 views

How is the SVM optimization objective derived from the hinge loss function?

The hinge loss function, in the context of SVMs, is given as: $$ \mathcal{L}(\mathbf{\vec w}, b\,; \mathbf{\vec x}^{(i)}, y ^{(i)}) = \max(0, 1-y ^{(i)}(\mathbf{\vec w}\cdot \mathbf{\vec x}^{(i)} + b))...
3 votes
2 answers
259 views

How do I check the validity of a set of inequality constraints? [closed]

I have a table of inequality constraints, each with an "x < y" relation. How can I check this table for contradictory logic such as a < b, b < c, c < a? For example ...
0 votes
0 answers
27 views

Robust or Stochastic Optimization Approach for Maximizing Profit with Limited Price Information

I am tackling a linear maximization problem where I need to select the optimal product among several options over a series of weeks, given certain constraints, in order to maximize future profit. The ...
4 votes
2 answers
125 views

What conditions are there on the exponent $p$ such that $\underset{\mu}{\arg\min}\left\{\mathbb E\left\vert X-\mu\right\vert^p\right\} $ must exist?

Let $X\sim F(x)$ be a (univariate) random variable defined by distribution function $F$. If the expected value exists, it is equal to $ \mathbb E[X] = \underset{\mu}{\arg\min}\left\{\mathbb E\left\...
0 votes
1 answer
244 views

Does coordinate wise convex function can be optimized more effectively?

I'm currently working on a non-convex function. It's basically a maximum likelihood problem so I'm trying to optimize this function. I know that non-convex optimization frequently reaches local optima ...
0 votes
0 answers
11 views

Hyperparameter optimization for CNN

I have a database of defect images on materials, like holes, cuts, and so on. There is not so much information inside the images, I am aware of it. I am using a CNN, in particular a ResNet50. I know ...
0 votes
0 answers
11 views

Using R to select individual resources that optimizes group effects to meet certain thresholds [closed]

See the following for a list of all constraints and references. Using R code, I would like to create an optimization that selects people (“A”,”B”,”C”) in order to maximize the number of products where ...
2 votes
1 answer
296 views

How does one recover the true solution to underdetermined equations when one has some prior or data about how the solution should look like?

I was interested in recovering the solution $x$ to a linear system underdetermined $N < D$: $$ Ax = y$$ as accurately as possible to the true $x$. Obviously, this system has infinite number of ...
-1 votes
0 answers
20 views

Why isn't ROC maximum for threshold 0.5?

the way I understand logistic regression, threshold = 0.5 basically produces a hyperplane to classify inputs which minimizes log loss (all of which is converted into a 0 to 1 range using sigmoid), so ...
6 votes
1 answer
4k views

What is a trust region reflective algorithm?

What is a trust region reflective algorithm? I know (from the matlab help) that it is used for solving constrained optimization problems. How is it different than the Levenberg-Marquardt algorithm ...
0 votes
0 answers
22 views

Scholkopf single class linear SVM equation: why ρ substracted to 1/2 ||w||² is the same as maximizing the distance

In the one class linear SVM, the equation is : $\min_{w, \rho} \frac{1}{2} \|w\|^2- \rho + C\sum_{i=1}^{n} \xi_i$ subject to: $\begin{align*} & w \cdot x_i \geq \rho - \xi_i, \\ & \xi_i \geq 0,...
1 vote
1 answer
242 views

Why the objective function in Fisher Discriminant Analysis?

I know FDA wants to find some linear combination $z = W^\top x$ so that the projected data has maximum between-class covariance and minimum within-class covariance. The first thing that came to my ...
0 votes
0 answers
4 views

Custom Model For Approximating Sin Function Using Backpropagation [duplicate]

I have very simple custom model which I am doing experiment with, I have model which takes one input and produce one output. the model equation is: y = sin(ax + b). (a) and (b) are single learnable ...
3 votes
1 answer
36 views

How to optimize a clinical scoring algorithm?

I've made two studies on clinical data that correlates with a disease. The clinical data can be aggregated into a score, such that the higher the score the higher your % of having the disease. However,...
1 vote
0 answers
31 views

Closed Form Solution for MLE parameter defining Linear Combination of two multivariate normal distributions

I have one set of $n$ observations which can be described as a single vector sampled from a multivariate normal distribution of the following form: $$ (1-\lambda)\mathbb{I}_n + \lambda \Sigma_{n} $$ ...
0 votes
1 answer
400 views

Constraints, bounds, and initialization variables in the GARCH / ARMA-GARCH models

I am interested in the correct way to estimate a GARCH/ARMA-GARCH model. I will refer to the coefficients as: ...
3 votes
1 answer
82 views

How to show the existence of global minimizer of Lasso type of objective function?

Suppose the objective function to be minimized is $$F(\theta) = \|y - X \theta\|_2^2 + \sum_{i=1}^p \lambda_i |\theta_i|$$ where $\theta$ is the independent variable which is feasible in $\mathbb{R}^p$...
0 votes
0 answers
18 views

Intercept term of logistic regression in ADMM algorithm

On page 66, the authors of article of ADMM says that the algorithm can be modified to obtain the intercept term easily in the sparse logistic regression model. Can someone explain this easy ...
0 votes
0 answers
39 views

Huber-Loss optimisation using Stochastic Gradient Descent to estimate intercept and coefficient of regression line

What: I'm trying to minimise the Huber-Loss for a linear regression using Stochastic Gradient Descent from scratch. Problem: It seems like that the coeffcient $m$ doesn't get optimised, therefore the ...
2 votes
0 answers
71 views

Do discontinuous functions have subgradients also?

Typically, the subgradient is defined for convex functions. And convex functions are continuous. However, DeepMind's VQ-VAE paper defines a model with a discontinuous vector quantization (VQ) layer, ...
1 vote
1 answer
462 views

How can I see a covariance matrix, as a standard regression y-Bx

ok so I have 2 assets, asset A and asset B, this assets have a vector of returns , 30 observations each. I calculate the estimated return as mean of the asset vector A and for B as a mean of asset ...
1 vote
1 answer
295 views

Least squares optimization with expensive model and many parameters

I have a physical model which takes $\sim50$ parameters and gives $\sim 2000$ outputs taking tens of minutes to run. I need to optimise these parameters to give outputs as close as possible to data by ...
2 votes
1 answer
44 views

Computing gradient over all examples in gradient descent

I am studying about Gradient Descent and Stochastic Gradient Descent, and the text says that one of the advantages of sgd over gd is, that gd can be computationally expensive for large datasets. In ...
11 votes
1 answer
254 views

Reconciling optimisation for log-likelihood and Brier score

Both log-likelihood and Brier score are proper scoring rules. As such, they reach the optimum when the predicted probabilities match the true ones. Since there is only one true probability for each ...
5 votes
1 answer
4k views

How to find initial values for Weibull MLE in R?

I want to find parameter estimates using MLE for a weibull distribution to some data: 604 104 224 200 1444 1076 1308 6084 468 2308. My code is as follows: ...
1 vote
1 answer
13 views

Gambling in multiple rounds with a maximum permitted bankroll and favorable or unfavourable probabilities

This is based on a deleted question, with the premises clarified to my understanding. You are gambling in a casino with particular rules: Bets are paid off at even amounts, so if you win a round you ...
2 votes
1 answer
211 views

How to maximize the ELBO in coordinate ascent variational inference

In the lecture by D.Blei: https://www.cs.princeton.edu/courses/archive/fall11/cos597C/lectures/variational-inference-i.pdf Variational inference is explained and he shows how to derive the optimal ...
1 vote
0 answers
22 views

Can we solve by hand the early exit multi-class classification problem? [closed]

Problem: Find a solution $\hat{\varepsilon}$ of the following minimization problem \begin{align*} &\min_{\varepsilon \in \mathbb{R}^M} \sum_{h=1}^M \varepsilon^h \hat{R}^h+\beta \sum_{h=1}^M \...
1 vote
1 answer
559 views

Optimization of a Convex Function involving Standard Normal CDF and PDF

Could someone provide closed form solutions, if any, and steps to get there for the following optimization problem? Please note this function has been shown to be a convex function and hence a minimum ...
0 votes
0 answers
19 views

Optimal Conditional Distribution for Minimising Information-Theoretic Expression

Consider two countable sets $\mathcal{X}$ and $\mathcal{Y}$. I aim to find the conditional distribution $P_{Y|X}$ that minimizes the following expression for any $x \in \mathcal{X}$ $$\sum_y P_{Y|X}(y|...
2 votes
1 answer
285 views

Finding optimal parameters for an unknown function - (learning with an unknown loss function)

I am working on a project where my objective is to find a set of parameters for a synthesizer which makes the synthesizer replicate an input sound. I do not know exactly what goes on inside of the ...
0 votes
1 answer
53 views

How to minimize a maximum of a function of 2 parameters with Pyhton [closed]

I calculate a distance between each point $c(x,y)$ and each point of $p(x,y)$. I need to find maximum among minimums of function: ...

1
2 3 4 5
56