Questions tagged [optimization]

Use this tag for any use of optimization within statistics.

Filter by
Sorted by
Tagged with
81 votes
6 answers
19k views

Optimization when Cost Function Slow to Evaluate

Gradient descent and many other methods are useful for finding local minima in cost functions. They can be efficient when the cost function can be evaluated quickly at each point, whether numerically ...
Jared Becksfort's user avatar
28 votes
2 answers
7k views

What is happening here, when I use squared loss in logistic regression setting?

I am trying to use squared loss to do binary classification on a toy data set. I am using mtcars data set, use mile per gallon and weight to predict transmission ...
Haitao Du's user avatar
  • 36.9k
57 votes
1 answer
27k views

Do we have to tune the number of trees in a random forest?

Software implementations of random forest classifiers have a number of parameters to allow users to fine-tune the algorithm's behavior, including the number of trees $T$ in the forest. Is this a ...
Sycorax's user avatar
  • 91k
7 votes
1 answer
3k views

How can change in cost function be positive?

In chapter 1 of Nielsen's Neural Networks and Deep Learning it says To make gradient descent work correctly, we need to choose the learning rate η to be small enough that Equation (9) is a good ...
fabiomaia's user avatar
  • 639
63 votes
6 answers
39k views

Practical hyperparameter optimization: Random vs. grid search

I'm currently going through Bengio's and Bergstra's Random Search for Hyper-Parameter Optimization [1] where the authors claim random search is more efficient than grid search in achieving ...
Bar's user avatar
  • 2,862
120 votes
6 answers
53k views

Is it possible to train a neural network without backpropagation?

Many neural network books and tutorials spend a lot of time on the backpropagation algorithm, which is essentially a tool to compute the gradient. Let's assume we are building a model with ~10K ...
Haitao Du's user avatar
  • 36.9k
28 votes
1 answer
9k views

Is there any intuitive explanation of why logistic regression will not work for perfect separation case? And why adding regularization will fix it?

We have many good discussions about perfect separation in logistic regression. Such as, Logistic regression in R resulted in perfect separation (Hauck-Donner phenomenon). Now what? and Logistic ...
Haitao Du's user avatar
  • 36.9k
252 votes
10 answers
127k views

Why is Newton's method not widely used in machine learning?

This is something that has been bugging me for a while, and I couldn't find any satisfactory answers online, so here goes: After reviewing a set of lectures on convex optimization, Newton's method ...
Fei Yang's user avatar
  • 2,511
84 votes
6 answers
45k views

What is an intuitive explanation for how PCA turns from a geometric problem (with distances) to a linear algebra problem (with eigenvectors)?

I've read a lot about PCA, including various tutorials and questions (such as this one, this one, this one, and this one). The geometric problem that PCA is trying to optimize is clear to me: PCA ...
stackoverflowuser2010's user avatar
10 votes
1 answer
6k views

Mean or sum of gradients for weight updates in SGD

I am using single observation to compute losses using neural network implementation in PyTorch. I am confused in a small detail of SGD. If I compute loss and do ...
pg2455's user avatar
  • 333
30 votes
3 answers
23k views

What are the impacts of choosing different loss functions in classification to approximate 0-1 loss

We know that some objective functions are easier to optimize and some are hard. And there are many loss functions that we want to use but hard to use, for example 0-1 loss. So we find some proxy loss ...
Haitao Du's user avatar
  • 36.9k
33 votes
3 answers
22k views

How could stochastic gradient descent save time compared to standard gradient descent?

Standard Gradient Descent would compute gradient for the entire training dataset. ...
Alina's user avatar
  • 1,165
11 votes
5 answers
7k views

Fitting SIR model with 2019-nCoV data doesn't conververge

I am trying to calculate the basic reproduction number $R_0$ of the new 2019-nCoV virus by fitting a SIR model to the current data. My code is based on https://arxiv.org/pdf/1605.01931.pdf, p. 11ff: <...
vonjd's user avatar
  • 6,156
151 votes
6 answers
172k views

Batch gradient descent versus stochastic gradient descent

Suppose we have some training set $(x_{(i)}, y_{(i)})$ for $i = 1, \dots, m$. Also suppose we run some type of supervised learning algorithm on the training set. Hypotheses are represented as $h_{\...
user20616's user avatar
  • 1,551
40 votes
1 answer
19k views

XGBoost Loss function Approximation With Taylor Expansion

As an example, take the objective function of the XGBoost model on the $t$'th iteration: $$\mathcal{L}^{(t)}=\sum_{i=1}^n\ell(y_i,\hat{y}_i^{(t-1)}+f_t(\mathbf{x}_i))+\Omega(f_t)$$ where $\ell$ is ...
Alex R.'s user avatar
  • 13.9k
30 votes
6 answers
11k views

For convex problems, does gradient in Stochastic Gradient Descent (SGD) always point at the global extreme value?

Given a convex cost function, using SGD for optimization, we will have a gradient (vector) at a certain point during the optimization process. My question is, given the point on the convex, does the ...
CyberPlayerOne's user avatar
19 votes
3 answers
7k views

Is PCA optimization convex?

The objective function of Principal Component Analysis (PCA) is minimizing the reconstruction error in L2 norm (see section 2.12 here. Another view is trying to maximize the variance on projection. We ...
Haitao Du's user avatar
  • 36.9k
67 votes
4 answers
104k views

Comparing SVM and logistic regression

Can someone please give me some intuition as to when to choose either SVM or LR? I want to understand the intuition behind what is the difference between the optimization criteria of learning the ...
user41799's user avatar
  • 721
39 votes
6 answers
71k views

Training a neural network for regression always predicts the mean

I am training a simple convolutional neural network for regression, where the task is to predict the (x,y) location of a box in an image, e.g.: The output of the network has two nodes, one for x, and ...
Karnivaurus's user avatar
  • 7,039
17 votes
5 answers
3k views

Philosophical question on logistic regression: why isn't the optimal threshold value trained?

Usually in logistic regression, we fit a model and get some predictions on the training set. We then cross-validate on those training predictions (something like here) and decide the optimal threshold ...
StatsSorceress's user avatar
5 votes
1 answer
1k views

What is the objective function to optimize in glm with gaussian and poisson family?

I am reading this post and still confused about the different ways of fitting exponential data. Specifically, why I am getting different results with following code? Could anyone help me to write down ...
Haitao Du's user avatar
  • 36.9k
29 votes
1 answer
11k views

Why using Newton's method for logistic regression optimization is called iterative re-weighted least squares?

Why using Newton's method for logistic regression optimization is called iterative re-weighted least squares? It seems not clear to me because logistic loss and least squares loss are completely ...
Haitao Du's user avatar
  • 36.9k
22 votes
1 answer
8k views

What is the connection between regularization and the method of lagrange multipliers ?

To prevent overfitting people people add a regularization term (proportional to the squared sum of the parameters of the model) with a regularization parameter $\lambda$ to the cost function of linear ...
asmaier's user avatar
  • 401
13 votes
2 answers
6k views

How to solve least absolute deviation by simplex method?

Here is the least absolute deviation problem under concerned: $ \underset{\textbf{w}}{\arg\min} L(w)=\sum_{i=1}^{n}|y_{i}-\textbf{w}^T\textbf{x}|$. I know it can be rearranged as LP problem in ...
southdoor's user avatar
  • 231
122 votes
3 answers
155k views

tanh activation function vs sigmoid activation function

The tanh activation function is: $$tanh \left( x \right) = 2 \cdot \sigma \left( 2 x \right) - 1$$ Where $\sigma(x)$, the sigmoid function, is defined as: $$\sigma(x) = \frac{e^x}{1 + e^x}$$. ...
satya's user avatar
  • 1,373
48 votes
1 answer
47k views

PCA objective function: what is the connection between maximizing variance and minimizing error?

The PCA algorithm can be formulated in terms of the correlation matrix (assume the data $X$ has already been normalized and we are only considering projection onto the first PC). The objective ...
Cam.Davidson.Pilon's user avatar
36 votes
5 answers
5k views

Can you overfit by training machine learning algorithms using CV/Bootstrap?

This question may well be too open-ended to get a definitive answer, but hopefully not. Machine learning algorithms, such as SVM, GBM, Random Forest etc, generally have some free parameters that, ...
Bogdanovist's user avatar
  • 6,619
25 votes
4 answers
9k views

Why are second-order derivatives useful in convex optimization?

I guess this is a basic question and it has to do with the direction of the gradient itself, but I'm looking for examples where 2nd order methods (e.g. BFGS) are more effective than simple gradient ...
Bar's user avatar
  • 2,862
20 votes
2 answers
6k views

Does log likelihood in GLM have guaranteed convergence to global maxima?

My questions are: Are generalized linear models (GLMs) guaranteed to converge to a global maximum? If so, why? Furthermore, what constraints are there on the link function to insure convexity? My ...
DankMasterDan's user avatar
12 votes
2 answers
15k views

Why is sum of squared residuals non-increasing when adding explanatory variable?

In my econometric textbook(Introductory Econometrics) covering OLS, the author write, "SSR must fall when another explanatory variable is added." Why is it?
Eric Xu's user avatar
  • 223
6 votes
1 answer
20k views

Fitting known equation to data

I have measured growth rates over a range of temperatures (temperature response curve) and would like to fit an already established equation/model to it. I'm very new to R and have trouble coding it ...
Kay's user avatar
  • 69
5 votes
1 answer
1k views

Why under joint least squares direction is it possible for some coefficients to decrease in LARS regression? [duplicate]

I think I understand how LARS regression works. It basically adds features to the model when they are more correlated with the residuals than the current model. ...
makansij's user avatar
  • 2,279
33 votes
2 answers
28k views

What is the difference between Maximum Likelihood Estimation & Gradient Descent?

What are the pro & cons of both the methods?
GeorgeOfTheRF's user avatar
25 votes
1 answer
4k views

Why can't a single ReLU learn a ReLU?

As a follow-up to My neural network can't even learn Euclidean distance I simplified even more and tried to train a single ReLU (with random weight) to a single ReLU. This is the simplest network ...
endolith's user avatar
  • 595
24 votes
2 answers
10k views

If p > n, the lasso selects at most n variables

One of the motivations for the elastic net was the following limitation of LASSO: In the $p > n$ case, the lasso selects at most n variables before it saturates, because of the nature of the ...
user1137731's user avatar
19 votes
2 answers
9k views

ARIMA estimation by hand

I'm trying to understand how the parameters are estimated in ARIMA modeling/Box Jenkins (BJ). Unfortunately none of the books that I have encountered describes the estimation procedure such as Log-...
forecaster's user avatar
  • 8,195
17 votes
3 answers
8k views

Software package to solve L-infinity norm linear regression

Is there any software package to solve the linear regression with the objective of minimizing the L-infinity norm.
Fan Zhang's user avatar
  • 409
15 votes
3 answers
3k views

Showing the Equivalence Between the $ {L}_{2} $ Norm Regularized Regression and $ {L}_{2} $ Norm Constrained Regression Using KKT

According to the references Book 1, Book 2 and paper. It has been mentioned that there is an equivalence between the regularized regression (Ridge, LASSO and Elastic Net) and their constraint ...
jeza's user avatar
  • 2,089
97 votes
7 answers
43k views

Why to optimize max log probability instead of probability

In most machine learning tasks where you can formulate some probability $p$ which should be maximised, we would actually optimize the log probability $\log p$ instead of the probability for some ...
Albert's user avatar
  • 1,245
52 votes
1 answer
39k views

How does the Adam method of stochastic gradient descent work?

I'm familiar with basic gradient descent algorithms for training neural networks. I've read the paper proposing Adam: ADAM: A METHOD FOR STOCHASTIC OPTIMIZATION. While I've definitely got some ...
daniel451's user avatar
  • 2,915
37 votes
5 answers
15k views

Why do smaller weights result in simpler models in regularization?

I completed Andrew Ng's Machine Learning course around a year ago, and am now writing my High School Math exploration on the workings of Logistic Regression and techniques to optimize on performance. ...
MCKapur's user avatar
  • 511
31 votes
2 answers
13k views

Why second order SGD convergence methods are unpopular for deep learning?

It seems that, especially for deep learning, there are dominating very simple methods for optimizing SGD convergence like ADAM - nice overview: http://ruder.io/optimizing-gradient-descent/ They trace ...
Jarek Duda's user avatar
28 votes
3 answers
13k views

Is Gradient Descent possible for kernelized SVMs (if so, why do people use Quadratic Programming)?

Why do people use Quadratic Programming techniques (such as SMO) when dealing with kernelized SVMs? What is wrong with Gradient Descent? Is it impossible to use with kernels or is it just too slow (...
MiniQuark's user avatar
  • 2,200
24 votes
3 answers
5k views

What causes sudden drops in training/test errors when training a neural network?

I've seen plots of test/training error suddenly dropping at certain epoch(s) a few times during the neural network training, and I wonder what causes these performance jumps: This image is taken from ...
libphy's user avatar
  • 341
22 votes
2 answers
1k views

How to choose between learning algorithms

I need to implement a program that will classify records into 2 categories (true/false) based on some training data, and I was wondering at which algorithm/methodology I should be looking at. There ...
Enno Shioji's user avatar
14 votes
3 answers
11k views

Step-by-step explanation of K-fold cross-validation with grid search to optimise hyperparameters

I'm well aware of the advantages of k-fold (and leave-one-out) cross-validation, as well as of the advantages of splitting your training set to create a third holdout 'validation' set, which you use ...
adb's user avatar
  • 143
8 votes
2 answers
2k views

Can sub-optimality of various hierarchical clustering methods be assessed or ranked?

Classic agglomerative hierarchical clustering methods are based on a greedy algorithm. This means that they (many of them) are prone to give sub-optimal solutions instead of the global optimum result, ...
ttnphns's user avatar
  • 57.6k
8 votes
1 answer
2k views

Equivalence between Elastic Net formulations

According to Hastie's paper, the elastic net has two equivalent formulations: $$\hat{\beta} = \underset{\beta}{\operatorname{argmin}} \left\{ \sum_{i=1}^N\left(y_i-\sum_{j=1}^p x_{ij} \beta_j\right)^...
skd's user avatar
  • 344
3 votes
6 answers
4k views

Why typically minimizing a cost instead of maximizing a reward?

I understand that, for example, maximizing the log-likelihood is equivalent to minimizing the negative log-likelihood. It is indeed a simple change, but still an extra step taken (it seems) for the ...
Julep's user avatar
  • 497
3 votes
1 answer
1k views

Graphical path Coordinate Descent in case of semi-differentiable functions such as Lasso

I am trying to understand how the graphical solution path to the optimum would look in the case of Lasso Regression. I can find only Pictures for the differentiable or non differentiable case. The ...
rook1996's user avatar
  • 465

1
2 3 4 5
8