Questions tagged [optimization]

Use this tag for any use of optimization within statistics.

Filter by
Sorted by
Tagged with
252 votes
10 answers
127k views

Why is Newton's method not widely used in machine learning?

This is something that has been bugging me for a while, and I couldn't find any satisfactory answers online, so here goes: After reviewing a set of lectures on convex optimization, Newton's method ...
Fei Yang's user avatar
  • 2,511
151 votes
6 answers
172k views

Batch gradient descent versus stochastic gradient descent

Suppose we have some training set $(x_{(i)}, y_{(i)})$ for $i = 1, \dots, m$. Also suppose we run some type of supervised learning algorithm on the training set. Hypotheses are represented as $h_{\...
user20616's user avatar
  • 1,551
122 votes
3 answers
155k views

tanh activation function vs sigmoid activation function

The tanh activation function is: $$tanh \left( x \right) = 2 \cdot \sigma \left( 2 x \right) - 1$$ Where $\sigma(x)$, the sigmoid function, is defined as: $$\sigma(x) = \frac{e^x}{1 + e^x}$$. ...
satya's user avatar
  • 1,373
120 votes
6 answers
53k views

Is it possible to train a neural network without backpropagation?

Many neural network books and tutorials spend a lot of time on the backpropagation algorithm, which is essentially a tool to compute the gradient. Let's assume we are building a model with ~10K ...
Haitao Du's user avatar
  • 36.9k
97 votes
7 answers
43k views

Why to optimize max log probability instead of probability

In most machine learning tasks where you can formulate some probability $p$ which should be maximised, we would actually optimize the log probability $\log p$ instead of the probability for some ...
Albert's user avatar
  • 1,245
84 votes
6 answers
45k views

What is an intuitive explanation for how PCA turns from a geometric problem (with distances) to a linear algebra problem (with eigenvectors)?

I've read a lot about PCA, including various tutorials and questions (such as this one, this one, this one, and this one). The geometric problem that PCA is trying to optimize is clear to me: PCA ...
stackoverflowuser2010's user avatar
81 votes
6 answers
19k views

Optimization when Cost Function Slow to Evaluate

Gradient descent and many other methods are useful for finding local minima in cost functions. They can be efficient when the cost function can be evaluated quickly at each point, whether numerically ...
Jared Becksfort's user avatar
77 votes
5 answers
88k views

What's the difference between momentum based gradient descent and Nesterov's accelerated gradient descent?

So momentum based gradient descent works as follows: $v=\beta m-\eta g$ where $m$ is the previous weight update, and $g$ is the current gradient with respect to the parameters $p$, $\eta$ is the ...
applecider's user avatar
  • 1,265
67 votes
4 answers
103k views

Comparing SVM and logistic regression

Can someone please give me some intuition as to when to choose either SVM or LR? I want to understand the intuition behind what is the difference between the optimization criteria of learning the ...
user41799's user avatar
  • 721
63 votes
6 answers
39k views

Practical hyperparameter optimization: Random vs. grid search

I'm currently going through Bengio's and Bergstra's Random Search for Hyper-Parameter Optimization [1] where the authors claim random search is more efficient than grid search in achieving ...
Bar's user avatar
  • 2,862
57 votes
1 answer
27k views

Do we have to tune the number of trees in a random forest?

Software implementations of random forest classifiers have a number of parameters to allow users to fine-tune the algorithm's behavior, including the number of trees $T$ in the forest. Is this a ...
Sycorax's user avatar
  • 91k
54 votes
1 answer
25k views

Understanding "almost all local minimum have very similar function value to the global optimum"

In a recent blog post by Rong Ge, it was said that: It is believed that for many problems including learning deep nets, almost all local minimum have very similar function value to the global ...
John Donn's user avatar
  • 641
52 votes
1 answer
39k views

How does the Adam method of stochastic gradient descent work?

I'm familiar with basic gradient descent algorithms for training neural networks. I've read the paper proposing Adam: ADAM: A METHOD FOR STOCHASTIC OPTIMIZATION. While I've definitely got some ...
daniel451's user avatar
  • 2,915
48 votes
1 answer
47k views

PCA objective function: what is the connection between maximizing variance and minimizing error?

The PCA algorithm can be formulated in terms of the correlation matrix (assume the data $X$ has already been normalized and we are only considering projection onto the first PC). The objective ...
Cam.Davidson.Pilon's user avatar
47 votes
1 answer
42k views

Neural Networks: weight change momentum and weight decay

Momentum $\alpha$ is used to diminish the fluctuations in weight changes over consecutive iterations: $$\Delta\omega_i(t+1) = - \eta\frac{\partial E}{\partial w_i} + \alpha \Delta \omega_i(t),$$ ...
Oleg Shirokikh's user avatar
43 votes
1 answer
24k views

Step-by-step example of reverse-mode automatic differentiation

Not sure if this question belongs here, but it's closely related to gradient methods in optimization, which seems to be on-topic here. Anyway, feel free to migrate if you think some other community ...
ffriend's user avatar
  • 9,990
43 votes
3 answers
22k views

When should one use Coordinate descent vs. gradient descent?

I was wondering what the different use cases are for the two algorithms, Coordinate Descent and Gradient Descent. I know that coordinate descent has problems with non-smooth functions but it is used ...
Bar's user avatar
  • 2,862
40 votes
1 answer
19k views

XGBoost Loss function Approximation With Taylor Expansion

As an example, take the objective function of the XGBoost model on the $t$'th iteration: $$\mathcal{L}^{(t)}=\sum_{i=1}^n\ell(y_i,\hat{y}_i^{(t-1)}+f_t(\mathbf{x}_i))+\Omega(f_t)$$ where $\ell$ is ...
Alex R.'s user avatar
  • 13.9k
39 votes
5 answers
14k views

Why use regularisation in polynomial regression instead of lowering the degree?

When doing regression, for example, two hyper parameters to choose are often the capacity of the function (eg. the largest exponent of a polynomial), and the amount of regularisation. What I'm ...
Karnivaurus's user avatar
  • 7,039
39 votes
4 answers
27k views

How should Feature Selection and Hyperparameter optimization be ordered in the machine learning pipeline?

My objective is to classify sensor signals. The concept of my solution so far is : i) Engineering features from raw signal ii) Selecting relevant features with ReliefF and a clustering approach iii) ...
Grunwalski's user avatar
39 votes
6 answers
71k views

Training a neural network for regression always predicts the mean

I am training a simple convolutional neural network for regression, where the task is to predict the (x,y) location of a box in an image, e.g.: The output of the network has two nodes, one for x, and ...
Karnivaurus's user avatar
  • 7,039
38 votes
1 answer
3k views

Why does glmer not achieve the maximum likelihood (as verified by applying further generic optimization)?

Numerically deriving the MLEs of GLMM is difficult and, in practice, I know, we should not use brute force optimization (e.g., using optim in a simple way). But for ...
quibble's user avatar
  • 1,436
37 votes
3 answers
23k views

Should training samples randomly drawn for mini-batch training neural nets be drawn without replacement?

We define an epoch as having gone through the entirety of all available training samples, and the mini-batch size as the number of samples over which we average to find the updates to weights/biases ...
phoenixdown's user avatar
37 votes
5 answers
15k views

Why do smaller weights result in simpler models in regularization?

I completed Andrew Ng's Machine Learning course around a year ago, and am now writing my High School Math exploration on the workings of Logistic Regression and techniques to optimize on performance. ...
MCKapur's user avatar
  • 511
36 votes
7 answers
26k views

Why are symmetric positive definite (SPD) matrices so important?

I know the definition of symmetric positive definite (SPD) matrix, but want to understand more. Why are they so important, intuitively? Here is what I know. What else? For a given data, Co-...
Haitao Du's user avatar
  • 36.9k
36 votes
5 answers
5k views

Can you overfit by training machine learning algorithms using CV/Bootstrap?

This question may well be too open-ended to get a definitive answer, but hopefully not. Machine learning algorithms, such as SVM, GBM, Random Forest etc, generally have some free parameters that, ...
Bogdanovist's user avatar
  • 6,619
36 votes
4 answers
30k views

Why is it important to include a bias correction term for the Adam optimizer for Deep Learning?

I was reading about the Adam optimizer for Deep Learning and came across the following sentence in the new book Deep Learning by Begnio, Goodfellow and Courtville: Adam includes bias corrections to ...
Charlie Parker's user avatar
36 votes
4 answers
49k views

How does batch size affect convergence of SGD and why?

I've seen similar conclusion from many discussions, that as the minibatch size gets larger the convergence of SGD actually gets harder/worse, for example this paper and this answer. Also I've heard of ...
dontloo's user avatar
  • 16.4k
34 votes
12 answers
9k views

The "Amazing Hidden Power" of Random Search?

I have the following question that compares random search optimization with gradient descent optimization: Based on the (amazing) answer provided over here Optimization when Cost Function Slow to ...
stats_noob's user avatar
34 votes
6 answers
22k views

Why study convex optimization for theoretical machine learning?

I am working on theoretical machine learning — on transfer learning, to be specific — for my Ph.D. Out of curiosity, why should I take a course on convex optimization? What take-aways from convex ...
Upendra01's user avatar
  • 1,946
33 votes
3 answers
10k views

Why does Bayesian Optimization perform poorly in more than 20 Dimensions?

I have been studying Bayesian Optimization lately and made the following notes about this topic: Unlike deterministic functions, real world functions are constructed using physical measurements ...
stats_noob's user avatar
33 votes
3 answers
22k views

How could stochastic gradient descent save time compared to standard gradient descent?

Standard Gradient Descent would compute gradient for the entire training dataset. ...
Alina's user avatar
  • 1,165
33 votes
2 answers
28k views

What is the difference between Maximum Likelihood Estimation & Gradient Descent?

What are the pro & cons of both the methods?
GeorgeOfTheRF's user avatar
32 votes
3 answers
5k views

Why Not Prune Your Neural Network?

Han et al. (2015) used a method of iterative pruning to reduce their network to only 10% of its original size with no loss of accuracy by removing weights with very low values, since these changed ...
RoryHector's user avatar
32 votes
1 answer
43k views

How does the L-BFGS work?

The purpose of the paper was to optimize some parameters by maximizing the regularized log-likelihood. Then they calculate Partial derivatives. And then authors mention that they optimize the equation ...
Abir's user avatar
  • 323
32 votes
2 answers
27k views

What is the reason that the Adam Optimizer is considered robust to the value of its hyper parameters?

I was reading about the Adam optimizer for Deep Learning and came across the following sentence in the new book Deep Learning by Bengio, Goodfellow and Courville: Adam is generally regarded as ...
Charlie Parker's user avatar
32 votes
1 answer
48k views

How to define the termination condition for gradient descent?

Actually, I wanted to ask you how can I define the terminating condition for gradient descent. Can I stop it based upon the number of iterations, i.e. considering parameter values for, say, 100 ...
user31820's user avatar
  • 1,501
31 votes
6 answers
4k views

Why not use the third derivative for numerical optimization?

If Hessians are so good for optimization (see e.g. Newton's method), why stop there? Let's use the third, fourth, fifth, and sixth derivatives? Why not?
echo's user avatar
  • 961
31 votes
2 answers
13k views

Why second order SGD convergence methods are unpopular for deep learning?

It seems that, especially for deep learning, there are dominating very simple methods for optimizing SGD convergence like ADAM - nice overview: http://ruder.io/optimizing-gradient-descent/ They trace ...
Jarek Duda's user avatar
30 votes
6 answers
11k views

For convex problems, does gradient in Stochastic Gradient Descent (SGD) always point at the global extreme value?

Given a convex cost function, using SGD for optimization, we will have a gradient (vector) at a certain point during the optimization process. My question is, given the point on the convex, does the ...
CyberPlayerOne's user avatar
30 votes
4 answers
8k views

Why maximum likelihood and not expected likelihood?

Why is it so common to obtain maximum likelihood estimates of parameters, but you virtually never hear about expected likelihood parameter estimates (i.e., based on the expected value rather than the ...
Jake Westfall's user avatar
30 votes
3 answers
23k views

What are the impacts of choosing different loss functions in classification to approximate 0-1 loss

We know that some objective functions are easier to optimize and some are hard. And there are many loss functions that we want to use but hard to use, for example 0-1 loss. So we find some proxy loss ...
Haitao Du's user avatar
  • 36.9k
30 votes
1 answer
2k views

What are the classical notations in statistics, linear algebra and machine learning? And what are the connections between these notations?

When we read a book, understanding the notations plays a very important role of understanding the contents. Unfortunately, different communities have different notation conventions for the formulation ...
29 votes
2 answers
44k views

What is the meaning of super script 2 subscript 2 within the context of norms?

I am new to optimization. I keep seeing equations that have a superscript 2 and a subscript 2 on the right-hand side of a norm. For instance, here is the least squares equation min $ ||Ax-b||^2_2$ I ...
bernie2436's user avatar
29 votes
1 answer
11k views

Why using Newton's method for logistic regression optimization is called iterative re-weighted least squares?

Why using Newton's method for logistic regression optimization is called iterative re-weighted least squares? It seems not clear to me because logistic loss and least squares loss are completely ...
Haitao Du's user avatar
  • 36.9k
29 votes
2 answers
29k views

Can gradient descent be applied to non-convex functions?

I'm just learning about optimization, and having trouble understanding the difference between convex and non-convex optimization. From my understanding, a convex function is one where "the line ...
Karnivaurus's user avatar
  • 7,039
29 votes
3 answers
8k views

Are line search methods used in deep learning? Why not?

A lot of tutorials online talk about gradient descent and almost all of them use a fixed step size (learning rate $\alpha$). Why is there no use of line search (such as backtracking line search or ...
Haitao Du's user avatar
  • 36.9k
29 votes
4 answers
16k views

When are genetic algorithms a good choice for optimization?

Genetic algorithms are one form of optimization method. Often stochastic gradient descent and its derivatives are the best choice for function optimization, but genetic algorithms are still sometimes ...
Gus's user avatar
  • 393
28 votes
5 answers
13k views

Is there always a maximizer for any MLE problem?

I wonder if there is always a maximizer for any maximum (log-)likelihood estimation problem? In other words, is there some distribution and some of its parameters, for which the MLE problem does not ...
Tim's user avatar
  • 19.5k
28 votes
2 answers
7k views

What is happening here, when I use squared loss in logistic regression setting?

I am trying to use squared loss to do binary classification on a toy data set. I am using mtcars data set, use mile per gallon and weight to predict transmission ...
Haitao Du's user avatar
  • 36.9k

1
2 3 4 5
56