# Questions tagged [optimization]

Use this tag for any use of optimization within statistics.

1,004 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
6k views

I've learned from DL classes that Adam should be the default choice for neural network training. However, I've recently seen more and more recent reinforcement learning agents use RMSProp instead of ...
• 1,075
754 views

• 2,686
171 views

### Compressed sensing: Optimization in $L_1$ norm and total variation with fourier coefficients

I'm reading the article Robust Uncertainty Principles: Exact Signal Reconstruction from Highly Incomplete Frequency Information (Candes, Romberg and Tao, 2004). In this article they are talking ...
• 829
118 views

### Local optima in high-dimensional optimization

I remember a theorem along the lines of In higher dimensional optimization problems, you are less likely to get stuck in local optima, because the more dimensions you have, the more likely you are to ...
• 367
2k views

### Momentum vs Polyak averaging

I'm going through this deck but don't quite get the difference between momentum and Polyak averaging, and what role Polyak averaging plays in modern optimizers. For example, is it correct to say that ...
• 4,458
206 views

### Does Fisher scoring always outperform Newton optimization?

My understanding is that Fisher scoring has several advantages over Newton raphson optimization such as Computational efficiency: if certain conditions are met (example:During MLE estimation, if link ...
• 323
157 views

### Does EM algorithm require us to know the joint (predictive) distribution of the latent variables $Z$ when $Z$ is two-dimensional?

In its general form the E-step of the EM algorithm finds the expectation $$Q(\theta|\theta') =\int \log[ p(Y,Z | \theta)] p(Z|Y,\theta') d Z$$ where $Y$ the data, $Z$ the latent variables, $\theta'$...
• 6,572
4k views

### How the Hessian matrix is used in optimization if you can't invert it

I've seen quite a lot of work to do with approximating the Hessian such as the Hessian Vector Product but I'm not entirely sure how knowing the Hessian helps us evaluate the gradient step to take. ...
945 views

### Box constraints with BFGS algorithm

I've been a long time adept of the Broyden-Fletcher-Goldfarb-Shanno algorithm (BFGS), which I trusted to be a pretty efficient local optimisation technique. And indeed it is. The problem I usually ...
• 1,546
621 views

I am looking for a review or comparison of global optimization techniques where the gradient of the function is available and utilized to speed up search, like the following: A hybrid descent method ...
• 983
4k views

### What is a trust region reflective algorithm?

What is a trust region reflective algorithm? I know (from the matlab help) that it is used for solving constrained optimization problems. How is it different than the Levenberg-Marquardt algorithm ...
113 views

### How are a set of chi squared statistics distributed?

I'm using a genetic algorithm to fit a system of ODE's to some data. Given the high dimentionality of the ODE model, the optimization problem is still a fairly difficult problem. Therefore I am ...
• 213
193 views

### Estimating parameters using Kullback-Leibler or Kolmogorov-Smirnoff via Nelder-Mead

I want to find the parameters of a model which specifies a set of classification probabilities, for say M classes. (I'll use the parameters in another model later.) Given a set of parameters $\theta$,...
• 389
263 views

### Examples in the Real World where Evolutionary Algorithms/Genetic Algorithms Outperform other Classes of Optimization Algorithms

I have been trying to do some research to find out if there are certain industries/types of problems or even specific examples in applied research paper where Evolutionary Algorithms (e.g. Genetic ...
706 views

### What are the advantages/disadvantages of design of experiments (DoE) versus stochastic optimization methods

I am working in a project to assist an experimental team in optimizing reaction conditions. The problem involves a large number of dimensions, i.e. 30+ reactants which we are trying out different ...
• 143
338 views

I have found mentions of two advantages in using gradients instead of actual residuals: 1) Using gradients will allow us to plug in any loss function (not just mse) without having to change our base ...
• 51
926 views

As is known to all, stochastic gradient descent is a popular optimizer in machine learning. There have been many variants of SGD. However, it has come to my attention that no one talks about the ...
• 151
1k views

### How does Canonical Time Warping help in time alignment?

Canonical Time Warping is a state-of-the-art technique for time alignment. According to the original paper, it helps account for individual varieties when aligning sequences derived from different ...
• 121
2k views

### Confusion about Robbins-Monro algorithm in Bishop PRML

This is basically how Robbins-Monro is presented in chapter 2.3 of Bishop's PRML book (from his slides): In the general update equation, $$\theta^{(N)} = \theta^{(N-1)} - \alpha_{N-1}z(θ^{(N-1)})$$ ...
• 682
2k views

### statsmodels: quantreg convergence cycle warning

I am getting the same Convergence cycle detected warning running a quantile regression with ...
• 1,198
542 views

• 1,523
970 views

### Analyse sensitivity of hyper-parameters of Machine Learning Models

I want to analyse how sensitive my non neural net machine learning models are to the choice of the different parameters. I am currently using grid search to tune the models. Is there any method that I ...
• 543
194 views

What is the most efficient way to solve linear Least absolute deviation regression problem? I know it can be solved using linear programming, is there a better/faster method? Edit: I'm interested ...
• 183
363 views

### Optimization textbooks for statistics and data analytics

Any statistical analysis, machine learning or data science involves some sort of optimization at the end of the day. I'm looking for good linear and nonlinear optimization textbooks for self ...
2k views

### Why does Nesterov momentum not improve the rate of convergence in the stochastic gradient case?

I have been reading Deep Learning book by Ian Goodfellow, where they wrote in chapter 8 (section 8.3.3) that Nesterov momentum does not improve the rate of convergence in stochastic gradient case. ...
• 601
86 views

### Automatic fitting of normalization constant as a parameter in noise contrastive estimation

In the paper on Noise Contrastive Estimation, the authors define a parameterized density function $p_m^0\left(x;\alpha\right)$ to estimate the unnormalized PDF of the data, and then further define a ...
• 1,461
1k views

• 2,164
303 views

### Machine Learning and Flow Maximization

Has anyone ever seen machine learning (ML) used to assist a Max Flow algorithm? I have a very large directed graph that has some fractal characteristics, meaning that this large graph can be roughly ...
• 41
61 views

### How to parameterize coefficient matrix to restrict eigenvalues?

Consider the $r-$dimensional autoregression $$y_t = Ay_{t-1} + v_t, v_t \overset{iid}{\sim}N(0,\Sigma).$$ It is well known that if all eigenvalues of $A$ have modulus less than unity then this ...
• 4,541
295 views

### Maximum likelihood estimation involving both probabilities and probability densities

Note: based on suggestions in the comments, I have rewritten this question. Please refer to the history for the original version. In general my question regards how to compute likelihoods in mixed ...
• 509