Questions tagged [optimization]

Use this tag for any use of optimization within statistics.

1,004 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
9 votes
0 answers
6k views

Advantage of RMSProp over Adam?

I've learned from DL classes that Adam should be the default choice for neural network training. However, I've recently seen more and more recent reinforcement learning agents use RMSProp instead of ...
Maybe's user avatar
  • 1,075
9 votes
0 answers
754 views

The Regularization Path for Smoothing Splines

I've got a potentially interesting question. Does anyone know if R already has a package for calculating the entire regularization path of the smoothing spline? That is, for: $$\hat{f}_{\lambda}=...
Nick Thieme's user avatar
  • 1,254
9 votes
0 answers
162 views

Territories from observations

I have a number of animal observations, and want to deduce the number of territories (i.e. the number of individual animals) from this. More formally, the problem can be stated as follows: Each ...
MvG's user avatar
  • 500
8 votes
0 answers
678 views

Are there any General Proofs on Genetic Algorithms?

Are there any general proofs or theorems relating to "genetic algorithms"? I have been reading about a theorem in math called the "Schema Theorem" - this theorem is one of the ...
stats_noob's user avatar
8 votes
0 answers
267 views

D-Optimal Criteria Vs Differential Shannon Entropy

How minimizing the determinant of the information matrix is equivalent to maximizing the differential Shannon entropy? A similar question was posted in Math SE but hasn't been rigorously answered. My ...
GENIVI-LEARNER's user avatar
8 votes
0 answers
631 views

How good is an optimal design (DOE)?

I am looking at an experiment with 5 factors (4 numeric and one nominal) with three levels for each numeric and 2 levels for the nominal. Instead of the 162 runs, I am interested in a small design (30 ...
B_Miner's user avatar
  • 8,640
8 votes
0 answers
192 views

Scope of non-linear least squares

edit: tl;dr: I can coerce a lot of optimization problems to take the form of a non-linear least squares problem, but does it make sense to do so? Suppose we have some empirical data $P=\{(x_i', y_i')\...
alang's user avatar
  • 323
7 votes
0 answers
5k views

Justification of acceptance probability in simulated annealing

In simulated annealing the acceptance probability for a new state in step $k$ is traditionally defined as $$ P(\text{accept new})= \begin{cases} \exp(-\frac{\Delta}{T_k}), & \text{ if } \Delta \...
g g's user avatar
  • 2,686
7 votes
0 answers
171 views

Compressed sensing: Optimization in $L_1$ norm and total variation with fourier coefficients

I'm reading the article Robust Uncertainty Principles: Exact Signal Reconstruction from Highly Incomplete Frequency Information (Candes, Romberg and Tao, 2004). In this article they are talking ...
Roy's user avatar
  • 829
6 votes
0 answers
118 views

Local optima in high-dimensional optimization

I remember a theorem along the lines of In higher dimensional optimization problems, you are less likely to get stuck in local optima, because the more dimensions you have, the more likely you are to ...
sheß's user avatar
  • 367
6 votes
1 answer
2k views

Momentum vs Polyak averaging

I'm going through this deck but don't quite get the difference between momentum and Polyak averaging, and what role Polyak averaging plays in modern optimizers. For example, is it correct to say that ...
Josh's user avatar
  • 4,458
6 votes
0 answers
206 views

Does Fisher scoring always outperform Newton optimization?

My understanding is that Fisher scoring has several advantages over Newton raphson optimization such as Computational efficiency: if certain conditions are met (example:During MLE estimation, if link ...
Preetham_tsp's user avatar
6 votes
0 answers
157 views

Does EM algorithm require us to know the joint (predictive) distribution of the latent variables $Z$ when $Z$ is two-dimensional?

In its general form the E-step of the EM algorithm finds the expectation $$ Q(\theta|\theta') =\int \log[ p(Y,Z | \theta)] p(Z|Y,\theta') d Z$$ where $Y$ the data, $Z$ the latent variables, $\theta'$...
tomka's user avatar
  • 6,572
6 votes
0 answers
4k views

How the Hessian matrix is used in optimization if you can't invert it

I've seen quite a lot of work to do with approximating the Hessian such as the Hessian Vector Product but I'm not entirely sure how knowing the Hessian helps us evaluate the gradient step to take. ...
tryingtolearn's user avatar
6 votes
0 answers
945 views

Box constraints with BFGS algorithm

I've been a long time adept of the Broyden-Fletcher-Goldfarb-Shanno algorithm (BFGS), which I trusted to be a pretty efficient local optimisation technique. And indeed it is. The problem I usually ...
Quantuple's user avatar
  • 1,546
6 votes
0 answers
621 views

Gradient-informed global optimization

I am looking for a review or comparison of global optimization techniques where the gradient of the function is available and utilized to speed up search, like the following: A hybrid descent method ...
user76284's user avatar
  • 983
6 votes
1 answer
4k views

What is a trust region reflective algorithm?

What is a trust region reflective algorithm? I know (from the matlab help) that it is used for solving constrained optimization problems. How is it different than the Levenberg-Marquardt algorithm ...
Learn_and_Share's user avatar
6 votes
0 answers
113 views

How are a set of chi squared statistics distributed?

I'm using a genetic algorithm to fit a system of ODE's to some data. Given the high dimentionality of the ODE model, the optimization problem is still a fairly difficult problem. Therefore I am ...
CiaranWelsh's user avatar
6 votes
0 answers
193 views

Estimating parameters using Kullback-Leibler or Kolmogorov-Smirnoff via Nelder-Mead

I want to find the parameters of a model which specifies a set of classification probabilities, for say M classes. (I'll use the parameters in another model later.) Given a set of parameters $\theta$,...
Yoda's user avatar
  • 389
5 votes
1 answer
263 views

Examples in the Real World where Evolutionary Algorithms/Genetic Algorithms Outperform other Classes of Optimization Algorithms

I have been trying to do some research to find out if there are certain industries/types of problems or even specific examples in applied research paper where Evolutionary Algorithms (e.g. Genetic ...
stats_noob's user avatar
5 votes
1 answer
706 views

What are the advantages/disadvantages of design of experiments (DoE) versus stochastic optimization methods

I am working in a project to assist an experimental team in optimizing reaction conditions. The problem involves a large number of dimensions, i.e. 30+ reactants which we are trying out different ...
Tianxun Zhou's user avatar
5 votes
0 answers
338 views

Why do we use gradients instead of residuals in Gradient Boosting?

I have found mentions of two advantages in using gradients instead of actual residuals: 1) Using gradients will allow us to plug in any loss function (not just mse) without having to change our base ...
eyio's user avatar
  • 51
5 votes
0 answers
926 views

Why no one talks about stochastic conjugate gradient descent?

As is known to all, stochastic gradient descent is a popular optimizer in machine learning. There have been many variants of SGD. However, it has come to my attention that no one talks about the ...
xixumei's user avatar
  • 151
5 votes
1 answer
1k views

How does Canonical Time Warping help in time alignment?

Canonical Time Warping is a state-of-the-art technique for time alignment. According to the original paper, it helps account for individual varieties when aligning sequences derived from different ...
Gene's user avatar
  • 121
5 votes
1 answer
2k views

Confusion about Robbins-Monro algorithm in Bishop PRML

This is basically how Robbins-Monro is presented in chapter 2.3 of Bishop's PRML book (from his slides): In the general update equation, $$ \theta^{(N)} = \theta^{(N-1)} - \alpha_{N-1}z(θ^{(N-1)}) $$ ...
Yibo Yang's user avatar
  • 682
5 votes
0 answers
2k views

statsmodels: quantreg convergence cycle warning

I am getting the same Convergence cycle detected warning running a quantile regression with ...
IcannotFixThis's user avatar
5 votes
0 answers
542 views

Learning hidden Markov model where transition/emission/initial probabilities aren't independent

I'm working on a problem that I've cast as an HMM, except that unlike the "traditional" case where the transition probabilities $a(i,j) = p(s_i = j \,|\, s_{i-1}=i)$, emission probabilities $b(j,o) = ...
Josh's user avatar
  • 59
4 votes
0 answers
36 views

Fused lasso for image denonising

For a given data $y_{i}$, with $i=1, \dots, n$, we consider the following signal approximation: $$ \hat{y} = \arg \min_{w}\sum_{i=1}^{n}(y_{i}-w_{i})^{2} + \lambda \sum_{(i,j)\in E}|w_{i} - w_{j}|, $$ ...
ABK's user avatar
  • 536
4 votes
0 answers
392 views

Is there any advantage from using Momentum Schedulers in training models using SGD than using a constant momentum of 0.9?

Recently I noticed that some pytorch repos of papers use Learning Rate Scheduler and momentum rate Scheduler , a lot of momentum rate schedulers exist similar to LR scheduler ranging from Lambda, ...
mutli-arm-bandit's user avatar
4 votes
0 answers
39 views

Which ML algorithms can be used to optimize a weighted quadratic loss function?

I want to solve the following optimization problem: $$ L = n^{-1} \sum^n_{i=1} w_i ( y_i - \tau(x_i))^2 $$ where $w_i \in \mathbb{R}^+$ weights, $y_i \in \mathbb{R}$ outcome data, $x_i$ features/...
tomka's user avatar
  • 6,572
4 votes
1 answer
56 views

Why do we handpick a specific loss for classification

I get the MLE log likelihood to get a "good loss", and that DL models have non convex losses, thus leading to have local minima and so on. However, my point is a bit different. Assume that ...
Alberto's user avatar
  • 1,137
4 votes
2 answers
124 views

MSE of correlations

These might be dumb questions but I am having trouble to wrap my head around of a particular problem. I have a sparse count matrix $G $ that I want to optimize which is $N \times p$. Also, I have ...
eonurk's user avatar
  • 93
4 votes
0 answers
146 views

Is gradient descent guaranteed to converge to a local minimum (if it doesn't diverge)?

A few times in the literature, I have seen it suggested that higher learning rates can be bad because gradient descent may approach the neighbourhood of a minimum, but then "bounce around" ...
Denziloe's user avatar
  • 1,103
4 votes
0 answers
94 views

Beneficial dimension for 2nd order modelling in SGD optimization?

There are currently mostly used first order methods in SGD optimizers, second order are often seen too costly as e.g. full Hessian has size $D^2$ in dimension $D$. But we don't need full Hessian - ...
Jarek Duda's user avatar
4 votes
0 answers
197 views

Pareto optimality in Metropolis sampling

In the Metropolis sampling algorithm, we have some function $f(x)$ proportional to a probability distribution $P(x)$. To generate a random walk with stationary distribution $P(x)$, we generate a ...
Justin Solomon's user avatar
4 votes
0 answers
1k views

Gradient descent versus fixed point iteration

Fixed-point iteration Say I have the iteration $$x^{(k+1)} \leftarrow x^{(k)} + \alpha f(x^{(k)})$$ to find $x^\ast$, the root of $f$, i.e. $f(x^\ast)=0$, where $f:(a,b) \to \mathbb{R}$, $\exists ...
moreblue's user avatar
  • 1,523
4 votes
0 answers
970 views

Analyse sensitivity of hyper-parameters of Machine Learning Models

I want to analyse how sensitive my non neural net machine learning models are to the choice of the different parameters. I am currently using grid search to tune the models. Is there any method that I ...
frank's user avatar
  • 543
4 votes
0 answers
194 views

Most efficient LAD solver

What is the most efficient way to solve linear Least absolute deviation regression problem? I know it can be solved using linear programming, is there a better/faster method? Edit: I'm interested ...
Meni's user avatar
  • 183
4 votes
0 answers
363 views

Optimization textbooks for statistics and data analytics

Any statistical analysis, machine learning or data science involves some sort of optimization at the end of the day. I'm looking for good linear and nonlinear optimization textbooks for self ...
4 votes
0 answers
2k views

Why does Nesterov momentum not improve the rate of convergence in the stochastic gradient case?

I have been reading Deep Learning book by Ian Goodfellow, where they wrote in chapter 8 (section 8.3.3) that Nesterov momentum does not improve the rate of convergence in stochastic gradient case. ...
samra irshad's user avatar
4 votes
0 answers
86 views

Automatic fitting of normalization constant as a parameter in noise contrastive estimation

In the paper on Noise Contrastive Estimation, the authors define a parameterized density function $p_m^0\left(x;\alpha\right)$ to estimate the unnormalized PDF of the data, and then further define a ...
JPJ's user avatar
  • 1,461
4 votes
0 answers
1k views

Maximum Likelihood of Wishart parameters

I'm having some difficulty in deriving the ML estimation of the parameters of a Wishart distribution. Given a set of matrices $\{W_1, W_2,\dots,W_N\} \in \mathbb{C}^{k\times k}$ for which $W_i \sim ...
aepound's user avatar
  • 161
4 votes
0 answers
460 views

How to use MCMC / gibbs sampling instead of an optimization algorithm ?

I've tried and implementend Factorization Machines with different loss functions and optimization algorithms (SGD , coordinate descent, adagrad, adadelta ...) and I've seen that it's possible to use ...
mlx's user avatar
  • 281
4 votes
0 answers
1k views

Maximum Prediction in Gaussian Process

A Gaussian process (GP) is defined as a collection of random variables with a joint Gaussian distribution (Rasmussen 2006). It is well known that given observations $\left \{ \mathbf{x},\mathbf{y}\...
Wis's user avatar
  • 2,164
4 votes
0 answers
303 views

Machine Learning and Flow Maximization

Has anyone ever seen machine learning (ML) used to assist a Max Flow algorithm? I have a very large directed graph that has some fractal characteristics, meaning that this large graph can be roughly ...
rafbrl's user avatar
  • 41
4 votes
0 answers
61 views

How to parameterize coefficient matrix to restrict eigenvalues?

Consider the $r-$dimensional autoregression $$ y_t = Ay_{t-1} + v_t, v_t \overset{iid}{\sim}N(0,\Sigma). $$ It is well known that if all eigenvalues of $A$ have modulus less than unity then this ...
KOE's user avatar
  • 4,541
4 votes
0 answers
295 views

Maximum likelihood estimation involving both probabilities and probability densities

Note: based on suggestions in the comments, I have rewritten this question. Please refer to the history for the original version. In general my question regards how to compute likelihoods in mixed ...
monade's user avatar
  • 509
4 votes
0 answers
952 views

Efficient Portfolio Optimization Through Simulation

Apologies in advance for the (possibly?) poor terminology as I'm a bit of a novice in the field. I was torn whether to ask this on stackoverflow or here, so hope its the right place. Anyway, my ...
psandersen's user avatar
3 votes
0 answers
125 views

Euclidean and geodesic distance have different gradients. Does mixing the two concepts impair triplet learning?

The triplet loss is defined by Florian Schroff, Dmitry Kalenichenko, James Philbin in "FaceNet: A Unified Embedding for Face Recognition and Clustering" as $$ \mathcal L = \sum_\mathcal T \...
Sycorax's user avatar
  • 91k
3 votes
0 answers
142 views

Solving the SVM Dual Problem

This toy problem was just thought of by me to get an better intuition for the SVM algorithm. Assume the following optimization problem: $$ L(w, b, \alpha)=\frac{1}{2}||w||^2 - \sum_{i}\alpha_i[y_i(\...
kklaw's user avatar
  • 515

1
2 3 4 5
21