# Questions tagged [optimization]

Use this tag for any use of optimization within statistics.

2,779 questions
Filter by
Sorted by
Tagged with
10k views

### Why does Bayesian Optimization perform poorly in more than 20 Dimensions?

I have been studying Bayesian Optimization lately and made the following notes about this topic: Unlike deterministic functions, real world functions are constructed using physical measurements ...
17k views

### How to set limits using constrOptim in R?

I am using constrOptim to minimize a log likelihood function for maximum likelihood estimation of parameters. I wish to set the bounds on my parameters, but to not understand the constrOptim ...
36 views

### multiple testing and p-hacking when choosing from many models

I'm trying to train a random forest model with a particle swarm optimization algorithm because my target function is not smooth and unknown to me, that is, in essence, this is search and training at ...
587 views

### permutation testing a nested cross validation SVM

Say that I used a nested cross validation to do SVM classification on an fMRI dataset with hyperparameter tuning ( using a linear or rbf kernel). The classification on my outer cross validation folds ...
1k views

### How does Canonical Time Warping help in time alignment?

Canonical Time Warping is a state-of-the-art technique for time alignment. According to the original paper, it helps account for individual varieties when aligning sequences derived from different ...
592 views

### Bayesian Regression- Expectation Maximization

In Bayesian regression, we have $y_i=x_i^{T}w+\epsilon_i$ where $w \sim \mathcal{N}(0,\alpha)$ and $\epsilon_i \sim \mathcal{N}(0,\frac{1}{\beta})$. Inference of $\alpha$ and $\beta$ is done by ...
127k views

### Why is Newton's method not widely used in machine learning?

This is something that has been bugging me for a while, and I couldn't find any satisfactory answers online, so here goes: After reviewing a set of lectures on convex optimization, Newton's method ...
11 views

### Is there room for finding a more efficient hybrid optimization problem, in the context of optimization algorithms for MLE?

Recently finished my statistical modelling class, but it only briefly touched on Maximum Likelihood Estimates and I thought it was an interesting topic, so I decided to go deeper in my own time. I ...
12 views

1 vote
14 views

### Bayesian Optimization using randomForest surrogate model in R language is taking a very longer time to complete [closed]

I am running a Bayesian Optimization to optimize an objective function where the difference between the predicted validation set and the mean of initial output of the dataset is kept to the bearest ...
265 views

### Examples in the Real World where Evolutionary Algorithms/Genetic Algorithms Outperform other Classes of Optimization Algorithms

I have been trying to do some research to find out if there are certain industries/types of problems or even specific examples in applied research paper where Evolutionary Algorithms (e.g. Genetic ...
39 views

The hinge loss function, in the context of SVMs, is given as: $$\mathcal{L}(\mathbf{\vec w}, b\,; \mathbf{\vec x}^{(i)}, y ^{(i)}) = \max(0, 1-y ^{(i)}(\mathbf{\vec w}\cdot \mathbf{\vec x}^{(i)} + b))... 3 votes 2 answers 259 views ### How do I check the validity of a set of inequality constraints? [closed] I have a table of inequality constraints, each with an "x < y" relation. How can I check this table for contradictory logic such as a < b, b < c, c < a? For example ... 0 votes 0 answers 27 views ### Robust or Stochastic Optimization Approach for Maximizing Profit with Limited Price Information I am tackling a linear maximization problem where I need to select the optimal product among several options over a series of weeks, given certain constraints, in order to maximize future profit. The ... 4 votes 2 answers 125 views ### What conditions are there on the exponent p such that \underset{\mu}{\arg\min}\left\{\mathbb E\left\vert X-\mu\right\vert^p\right\}  must exist? Let X\sim F(x) be a (univariate) random variable defined by distribution function F. If the expected value exists, it is equal to  \mathbb E[X] = \underset{\mu}{\arg\min}\left\{\mathbb E\left\... 0 votes 1 answer 244 views ### Does coordinate wise convex function can be optimized more effectively? I'm currently working on a non-convex function. It's basically a maximum likelihood problem so I'm trying to optimize this function. I know that non-convex optimization frequently reaches local optima ... 0 votes 0 answers 11 views ### Hyperparameter optimization for CNN I have a database of defect images on materials, like holes, cuts, and so on. There is not so much information inside the images, I am aware of it. I am using a CNN, in particular a ResNet50. I know ... 0 votes 0 answers 11 views ### Using R to select individual resources that optimizes group effects to meet certain thresholds [closed] See the following for a list of all constraints and references. Using R code, I would like to create an optimization that selects people (“A”,”B”,”C”) in order to maximize the number of products where ... 2 votes 1 answer 296 views ### How does one recover the true solution to underdetermined equations when one has some prior or data about how the solution should look like? I was interested in recovering the solution x to a linear system underdetermined N < D:$$ Ax = yas accurately as possible to the true x. Obviously, this system has infinite number of ... -1 votes 0 answers 20 views ### Why isn't ROC maximum for threshold 0.5? the way I understand logistic regression, threshold = 0.5 basically produces a hyperplane to classify inputs which minimizes log loss (all of which is converted into a 0 to 1 range using sigmoid), so ... 6 votes 1 answer 4k views ### What is a trust region reflective algorithm? What is a trust region reflective algorithm? I know (from the matlab help) that it is used for solving constrained optimization problems. How is it different than the Levenberg-Marquardt algorithm ... 0 votes 0 answers 22 views ### Scholkopf single class linear SVM equation: why ρ substracted to 1/2 ||w||² is the same as maximizing the distance In the one class linear SVM, the equation is : \min_{w, \rho} \frac{1}{2} \|w\|^2- \rho + C\sum_{i=1}^{n} \xi_i subject to: \begin{align*} & w \cdot x_i \geq \rho - \xi_i, \\ & \xi_i \geq 0,... 1 vote 1 answer 242 views ### Why the objective function in Fisher Discriminant Analysis? I know FDA wants to find some linear combination z = W^\top x so that the projected data has maximum between-class covariance and minimum within-class covariance. The first thing that came to my ... 0 votes 0 answers 4 views ### Custom Model For Approximating Sin Function Using Backpropagation [duplicate] I have very simple custom model which I am doing experiment with, I have model which takes one input and produce one output. the model equation is: y = sin(ax + b). (a) and (b) are single learnable ... 3 votes 1 answer 36 views ### How to optimize a clinical scoring algorithm? I've made two studies on clinical data that correlates with a disease. The clinical data can be aggregated into a score, such that the higher the score the higher your % of having the disease. However,... 1 vote 0 answers 31 views ### Closed Form Solution for MLE parameter defining Linear Combination of two multivariate normal distributions I have one set of n observations which can be described as a single vector sampled from a multivariate normal distribution of the following form: (1-\lambda)\mathbb{I}_n + \lambda \Sigma_{n} $$... 0 votes 1 answer 400 views ### Constraints, bounds, and initialization variables in the GARCH / ARMA-GARCH models I am interested in the correct way to estimate a GARCH/ARMA-GARCH model. I will refer to the coefficients as: ... 3 votes 1 answer 82 views ### How to show the existence of global minimizer of Lasso type of objective function? Suppose the objective function to be minimized is$$F(\theta) = \|y - X \theta\|_2^2 + \sum_{i=1}^p \lambda_i |\theta_i|where \theta is the independent variable which is feasible in \mathbb{R}^p... 0 votes 0 answers 18 views ### Intercept term of logistic regression in ADMM algorithm On page 66, the authors of article of ADMM says that the algorithm can be modified to obtain the intercept term easily in the sparse logistic regression model. Can someone explain this easy ... 0 votes 0 answers 39 views ### Huber-Loss optimisation using Stochastic Gradient Descent to estimate intercept and coefficient of regression line What: I'm trying to minimise the Huber-Loss for a linear regression using Stochastic Gradient Descent from scratch. Problem: It seems like that the coeffcient m doesn't get optimised, therefore the ... 2 votes 0 answers 71 views ### Do discontinuous functions have subgradients also? Typically, the subgradient is defined for convex functions. And convex functions are continuous. However, DeepMind's VQ-VAE paper defines a model with a discontinuous vector quantization (VQ) layer, ... 1 vote 1 answer 462 views ### How can I see a covariance matrix, as a standard regression y-Bx ok so I have 2 assets, asset A and asset B, this assets have a vector of returns , 30 observations each. I calculate the estimated return as mean of the asset vector A and for B as a mean of asset ... 1 vote 1 answer 295 views ### Least squares optimization with expensive model and many parameters I have a physical model which takes \sim50 parameters and gives \sim 2000 outputs taking tens of minutes to run. I need to optimise these parameters to give outputs as close as possible to data by ... 2 votes 1 answer 44 views ### Computing gradient over all examples in gradient descent I am studying about Gradient Descent and Stochastic Gradient Descent, and the text says that one of the advantages of sgd over gd is, that gd can be computationally expensive for large datasets. In ... 11 votes 1 answer 254 views ### Reconciling optimisation for log-likelihood and Brier score Both log-likelihood and Brier score are proper scoring rules. As such, they reach the optimum when the predicted probabilities match the true ones. Since there is only one true probability for each ... 5 votes 1 answer 4k views ### How to find initial values for Weibull MLE in R? I want to find parameter estimates using MLE for a weibull distribution to some data: 604 104 224 200 1444 1076 1308 6084 468 2308. My code is as follows: ... 1 vote 1 answer 13 views ### Gambling in multiple rounds with a maximum permitted bankroll and favorable or unfavourable probabilities This is based on a deleted question, with the premises clarified to my understanding. You are gambling in a casino with particular rules: Bets are paid off at even amounts, so if you win a round you ... 2 votes 1 answer 211 views ### How to maximize the ELBO in coordinate ascent variational inference In the lecture by D.Blei: https://www.cs.princeton.edu/courses/archive/fall11/cos597C/lectures/variational-inference-i.pdf Variational inference is explained and he shows how to derive the optimal ... 1 vote 0 answers 22 views ### Can we solve by hand the early exit multi-class classification problem? [closed] Problem: Find a solution \hat{\varepsilon} of the following minimization problem \begin{align*} &\min_{\varepsilon \in \mathbb{R}^M} \sum_{h=1}^M \varepsilon^h \hat{R}^h+\beta \sum_{h=1}^M \... 1 vote 1 answer 559 views ### Optimization of a Convex Function involving Standard Normal CDF and PDF Could someone provide closed form solutions, if any, and steps to get there for the following optimization problem? Please note this function has been shown to be a convex function and hence a minimum ... 0 votes 0 answers 19 views ### Optimal Conditional Distribution for Minimising Information-Theoretic Expression Consider two countable sets \mathcal{X} and \mathcal{Y}. I aim to find the conditional distribution P_{Y|X} that minimizes the following expression for any x \in \mathcal{X}\sum_y P_{Y|X}(y|...
I calculate a distance between each point $c(x,y)$ and each point of $p(x,y)$. I need to find maximum among minimums of function: ...