Questions tagged [optimization]
Use this tag for any use of optimization within statistics.
2,800
questions
254
votes
10
answers
129k
views
Why is Newton's method not widely used in machine learning?
This is something that has been bugging me for a while, and I couldn't find any satisfactory answers online, so here goes:
After reviewing a set of lectures on convex optimization, Newton's method ...
- 2,531
153
votes
7
answers
174k
views
Batch gradient descent versus stochastic gradient descent
Suppose we have some training set $(x_{(i)}, y_{(i)})$ for $i = 1, \dots, m$. Also suppose we run some type of supervised learning algorithm on the training set. Hypotheses are represented as $h_{\...
- 1,571
122
votes
3
answers
157k
views
tanh activation function vs sigmoid activation function
The tanh activation function is:
$$\tanh x= 2 \cdot \sigma \left( 2 x \right) - 1$$
Where $\sigma(x)$, the sigmoid function, is defined as:
$$\sigma(x) = \frac{e^x}{1 + e^x}$$.
Questions:
Does it ...
- 1,373
121
votes
6
answers
54k
views
Is it possible to train a neural network without backpropagation?
Many neural network books and tutorials spend a lot of time on the backpropagation algorithm, which is essentially a tool to compute the gradient.
Let's assume we are building a model with ~10K ...
- 37.2k
98
votes
7
answers
44k
views
Why to optimize max log probability instead of probability
In most machine learning tasks where you can formulate some probability $p$ which should be maximised, we would actually optimize the log probability $\log p$ instead of the probability for some ...
- 1,265
85
votes
6
answers
46k
views
What is an intuitive explanation for how PCA turns from a geometric problem (with distances) to a linear algebra problem (with eigenvectors)?
I've read a lot about PCA, including various tutorials and questions (such as this one, this one, this one, and this one).
The geometric problem that PCA is trying to optimize is clear to me: PCA ...
- 3,600
83
votes
6
answers
20k
views
Optimization when Cost Function Slow to Evaluate
Gradient descent and many other methods are useful for finding local minima in cost functions. They can be efficient when the cost function can be evaluated quickly at each point, whether numerically ...
- 1,063
77
votes
5
answers
90k
views
What's the difference between momentum based gradient descent and Nesterov's accelerated gradient descent?
So momentum based gradient descent works as follows:
$v=\beta m-\eta g$
where $m$ is the previous weight update, and $g$ is the current gradient with respect to the parameters $p$, $\eta$ is the ...
- 1,265
68
votes
4
answers
105k
views
Comparing SVM and logistic regression
Can someone please give me some intuition as to when to choose either SVM or LR? I want to understand the intuition behind what is the difference between the optimization criteria of learning the ...
- 731
63
votes
6
answers
40k
views
Practical hyperparameter optimization: Random vs. grid search
I'm currently going through Bengio's and Bergstra's Random Search for Hyper-Parameter Optimization [1] where the authors claim random search is more efficient than grid search in achieving ...
- 2,872
59
votes
1
answer
28k
views
Do we have to tune the number of trees in a random forest?
Software implementations of random forest classifiers have a number of parameters to allow users to fine-tune the algorithm's behavior, including the number of trees $T$ in the forest. Is this a ...
- 92.5k
55
votes
2
answers
25k
views
Understanding "almost all local minimum have very similar function value to the global optimum"
In a recent blog post by Rong Ge, it was said that:
It is believed that for many problems including learning deep nets, almost all local minimum have very similar function value to the global ...
- 651
52
votes
1
answer
40k
views
How does the Adam method of stochastic gradient descent work?
I'm familiar with basic gradient descent algorithms for training neural networks. I've read the paper proposing Adam: ADAM: A METHOD FOR STOCHASTIC OPTIMIZATION.
While I've definitely got some ...
- 2,925
48
votes
1
answer
48k
views
PCA objective function: what is the connection between maximizing variance and minimizing error?
The PCA algorithm can be formulated in terms of the correlation matrix (assume the data $X$ has already been normalized and we are only considering projection onto the first PC). The objective ...
- 12.3k
47
votes
1
answer
42k
views
Neural Networks: weight change momentum and weight decay
Momentum $\alpha$ is used to diminish the fluctuations in weight changes over consecutive iterations:
$$\Delta\omega_i(t+1) = - \eta\frac{\partial E}{\partial w_i} + \alpha \Delta \omega_i(t),$$
...
- 895
44
votes
3
answers
23k
views
When should one use Coordinate descent vs. gradient descent?
I was wondering what the different use cases are for the two algorithms, Coordinate Descent and Gradient Descent.
I know that coordinate descent has problems with non-smooth functions but it is used ...
- 2,872
43
votes
1
answer
24k
views
Step-by-step example of reverse-mode automatic differentiation
Not sure if this question belongs here, but it's closely related to gradient methods in optimization, which seems to be on-topic here. Anyway, feel free to migrate if you think some other community ...
- 10k
42
votes
1
answer
20k
views
XGBoost Loss function Approximation With Taylor Expansion
As an example, take the objective function of the XGBoost model on the $t$'th iteration:
$$\mathcal{L}^{(t)}=\sum_{i=1}^n\ell(y_i,\hat{y}_i^{(t-1)}+f_t(\mathbf{x}_i))+\Omega(f_t)$$
where $\ell$ is ...
- 14k
41
votes
4
answers
28k
views
How should Feature Selection and Hyperparameter optimization be ordered in the machine learning pipeline?
My objective is to classify sensor signals.
The concept of my solution so far is :
i) Engineering features from raw signal
ii) Selecting relevant features with ReliefF and a clustering approach
iii) ...
- 665
39
votes
5
answers
15k
views
Why use regularisation in polynomial regression instead of lowering the degree?
When doing regression, for example, two hyper parameters to choose are often the capacity of the function (eg. the largest exponent of a polynomial), and the amount of regularisation. What I'm ...
- 7,109
39
votes
6
answers
74k
views
Training a neural network for regression always predicts the mean
I am training a simple convolutional neural network for regression, where the task is to predict the (x,y) location of a box in an image, e.g.:
The output of the network has two nodes, one for x, and ...
- 7,109
38
votes
5
answers
15k
views
Why do smaller weights result in simpler models in regularization?
I completed Andrew Ng's Machine Learning course around a year ago, and am now writing my High School Math exploration on the workings of Logistic Regression and techniques to optimize on performance. ...
- 521
38
votes
1
answer
3k
views
Why does glmer not achieve the maximum likelihood (as verified by applying further generic optimization)?
Numerically deriving the MLEs of GLMM is difficult and, in practice, I know, we should not use brute force optimization (e.g., using optim in a simple way). But for ...
- 1,694
37
votes
4
answers
32k
views
Why is it important to include a bias correction term for the Adam optimizer for Deep Learning?
I was reading about the Adam optimizer for Deep Learning and came across the following sentence in the new book Deep Learning by Begnio, Goodfellow and Courtville:
Adam includes bias corrections to ...
- 6,986
37
votes
3
answers
23k
views
Should training samples randomly drawn for mini-batch training neural nets be drawn without replacement?
We define an epoch as having gone through the entirety of all available training samples, and the mini-batch size as the number of samples over which we average to find the updates to weights/biases ...
- 661
36
votes
7
answers
27k
views
Why are symmetric positive definite (SPD) matrices so important?
I know the definition of symmetric positive definite (SPD) matrix, but want to understand more.
Why are they so important, intuitively?
Here is what I know. What else?
For a given data, Co-...
- 37.2k
36
votes
5
answers
5k
views
Can you overfit by training machine learning algorithms using CV/Bootstrap?
This question may well be too open-ended to get a definitive answer, but hopefully not.
Machine learning algorithms, such as SVM, GBM, Random Forest etc, generally have some free parameters that, ...
- 6,649
36
votes
4
answers
49k
views
How does batch size affect convergence of SGD and why?
I've seen similar conclusion from many discussions, that as the minibatch size gets larger the convergence of SGD actually gets harder/worse, for example this paper and this answer. Also I've heard of ...
- 16.7k
34
votes
12
answers
9k
views
The "Amazing Hidden Power" of Random Search?
I have the following question that compares random search optimization with gradient descent optimization:
Based on the (amazing) answer provided over here Optimization when Cost Function Slow to ...
34
votes
6
answers
23k
views
Why study convex optimization for theoretical machine learning?
I am working on theoretical machine learning — on transfer learning, to be specific — for my Ph.D.
Out of curiosity, why should I take a course on convex optimization?
What take-aways from convex ...
- 1,956
34
votes
4
answers
11k
views
Why does Bayesian Optimization perform poorly in more than 20 Dimensions?
I have been studying Bayesian Optimization lately and made the following notes about this topic:
Unlike deterministic functions, real world functions are constructed using physical measurements
...
33
votes
1
answer
45k
views
How does the L-BFGS work?
The purpose of the paper was to optimize some parameters by maximizing the regularized log-likelihood. Then they calculate Partial derivatives.
And then authors mention that they optimize the equation ...
- 333
33
votes
3
answers
22k
views
How could stochastic gradient descent save time compared to standard gradient descent?
Standard Gradient Descent would compute gradient for the entire training dataset.
...
- 1,175
33
votes
2
answers
29k
views
What is the difference between Maximum Likelihood Estimation & Gradient Descent?
What are the pro & cons of both the methods?
- 5,653
33
votes
2
answers
13k
views
Why second order SGD convergence methods are unpopular for deep learning?
It seems that, especially for deep learning, there are dominating very simple methods for optimizing SGD convergence like ADAM - nice overview: http://ruder.io/optimizing-gradient-descent/
They trace ...
- 431
33
votes
2
answers
27k
views
What is the reason that the Adam Optimizer is considered robust to the value of its hyper parameters?
I was reading about the Adam optimizer for Deep Learning and came across the following sentence in the new book Deep Learning by Bengio, Goodfellow and Courville:
Adam is generally regarded as ...
- 6,986
32
votes
3
answers
5k
views
Why Not Prune Your Neural Network?
Han et al. (2015) used a method of iterative pruning to reduce their network to only 10% of its original size with no loss of accuracy by removing weights with very low values, since these changed ...
- 443
32
votes
2
answers
29k
views
Can gradient descent be applied to non-convex functions?
I'm just learning about optimization, and having trouble understanding the difference between convex and non-convex optimization. From my understanding, a convex function is one where "the line ...
- 7,109
32
votes
1
answer
48k
views
How to define the termination condition for gradient descent?
Actually, I wanted to ask you how can I define the terminating condition for gradient descent.
Can I stop it based upon the number of iterations, i.e. considering parameter values for, say, 100 ...
- 1,511
31
votes
6
answers
4k
views
Why not use the third derivative for numerical optimization?
If Hessians are so good for optimization (see e.g. Newton's method), why stop there? Let's use the third, fourth, fifth, and sixth derivatives? Why not?
- 971
30
votes
6
answers
11k
views
For convex problems, does gradient in Stochastic Gradient Descent (SGD) always point at the global extreme value?
Given a convex cost function, using SGD for optimization, we will have a gradient (vector) at a certain point during the optimization process.
My question is, given the point on the convex, does the ...
- 2,475
30
votes
2
answers
45k
views
What is the meaning of super script 2 subscript 2 within the context of norms?
I am new to optimization. I keep seeing equations that have a superscript 2 and a subscript 2 on the right-hand side of a norm. For instance, here is the least squares equation
min $ ||Ax-b||^2_2$
I ...
- 423
30
votes
4
answers
9k
views
Why maximum likelihood and not expected likelihood?
Why is it so common to obtain maximum likelihood estimates of parameters, but you virtually never hear about expected likelihood parameter estimates (i.e., based on the expected value rather than the ...
- 12.8k
30
votes
3
answers
23k
views
What are the impacts of choosing different loss functions in classification to approximate 0-1 loss
We know that some objective functions are easier to optimize and some are hard. And there are many loss functions that we want to use but hard to use, for example 0-1 loss. So we find some proxy loss ...
- 37.2k
30
votes
1
answer
2k
views
What are the classical notations in statistics, linear algebra and machine learning? And what are the connections between these notations?
When we read a book, understanding the notations plays a very important role of understanding the contents. Unfortunately, different communities have different notation conventions for the formulation ...
29
votes
2
answers
12k
views
Why using Newton's method for logistic regression optimization is called iterative re-weighted least squares?
Why using Newton's method for logistic regression optimization is called iterative re-weighted least squares?
It seems not clear to me because logistic loss and least squares loss are completely ...
- 37.2k
29
votes
3
answers
8k
views
Are line search methods used in deep learning? Why not?
A lot of tutorials online talk about gradient descent and almost all of them use a fixed step size (learning rate $\alpha$). Why is there no use of line search (such as backtracking line search or ...
- 37.2k
29
votes
4
answers
7k
views
How to ensure properties of covariance matrix when fitting multivariate normal model using maximum likelihood?
Suppose I have the following model
$$y_i=f(x_i,\theta)+\varepsilon_i$$
where $y_i\in \mathbb{R}^K$ , $x_i$ is a vector of explanatory variables, $\theta$ is the parameters of non-linear function $...
- 35.2k
29
votes
4
answers
17k
views
When are genetic algorithms a good choice for optimization?
Genetic algorithms are one form of optimization method. Often stochastic gradient descent and its derivatives are the best choice for function optimization, but genetic algorithms are still sometimes ...
- 393
29
votes
3
answers
13k
views
Is Gradient Descent possible for kernelized SVMs (if so, why do people use Quadratic Programming)?
Why do people use Quadratic Programming techniques (such as SMO) when dealing with kernelized SVMs? What is wrong with Gradient Descent? Is it impossible to use with kernels or is it just too slow (...
- 2,220