The tag has no usage guidance.

159 questions
Filter by
Sorted by
Tagged with
14 views

### SGD with momentum update - CS231n explanation

In the notes for Stanford's CS231n course there is an explanation for the Momentum update. I'm confused by the usage of the word "integrates" here, e.g. "gradient directly integrates ...
1k views

### For regression with time varying parameters, SGD or Kalman filter?

What is the advantage of kalman filters as an online update mechanism instead of the stochastic gradient descent?
131k views

10 views

### Why do we need more than 1 epoch to train data? [duplicate]

As 1 epoch means each data point has gone through the algorithm once and made changes in weighted values accordingly . So , why there is a need to process same data again and again ? How does it ...
129 views

I'm reading Hands-On Machine Learning with Scikit-Learn, Keras & Tensorflow and on page 325 (follows up on 326) there's a following piece of text on learning-rate: The learning is arguably the ...
59 views

### SGD is sensitive to feature scaling

I am taking a deep learning class and the class slides state one of SGD's problems as: "Gradient is scaled equally across all dimensions." Now what is meant by this is I believe, when we ...
253 views

### SGD for Gaussian Process estimation

Given a Gaussian process with kernel function $K_{\theta}$ depending on some hyperparameters $\theta$ and set of observations $\{(x_i,y_i)\}_{i=1}^n$, I want to choose $\theta$ to maximize the ...
21 views

### Can a neural network still manage to converge, with slightly incorrect gradients?

In a network, we find gradients of the error function w.r.t each of the parameters used in the network. We then update the weights say, using vanilla Gradient Descent. If the computed gradients, do ...
791 views

### Is a loss function computed after each step of gradient descent or after a whole epoch?

In neural networks with mini-batch or stochastic gradient descent, is a loss function computed after each step of gradient descent or after a whole epoch?
56 views

### Optimization of Linear Autoencoder with SGD

I'm interested in the Linear Autoencoder(LAE), and I knew that, at convergence point, the subspace LAE learns is the same as the subspace PCA learns up to linear transformations. Also, the loss ...
1k views

I am using single observation to compute losses using neural network implementation in PyTorch. I am confused in a small detail of SGD. If I compute loss and do ...
15 views

### can alternating optimization be performed in mini-batches

Just wondering if alternating minimization could be performed in mini-batches (just like we have gradient descent and its mini-batch version). Although I am perfectly fine with the full batch version ...
78 views

146 views

### Problems, which are difficult for SGD

I am doing some research on problems, for which the stochastic gradient descent doesn't perform well. Often SGD is mentioned as the best method for the training of neural networks. However, I've also ...
44 views

### How does scaled conjugate gradient work in neural network training? Comparison with gradient descent

I am very new and beginner in the machine learning world, and I would like to ask if someone could simply explain to me how does the scaled conjugate gradient method work in neural network training? ...
4k views

### Why second order SGD convergence methods are unpopular for deep learning?

It seems that, especially for deep learning, there are dominating very simple methods for optimizing SGD convergence like ADAM - nice overview: http://ruder.io/optimizing-gradient-descent/ They trace ...
1k views

### The actual role of second-order optimization as oppose to first-order optimizations

I do not fully understand how second-order optimization approaches help machine learning algorithms, like multilayer perceptron, to achieve the global minimum error....
61 views

### Why not use line search in conjunction with stochastic gradient descent?

I'm familiar with numerical optimization in Engineering context. I have taken several graduate level engineering optimization and operations research courses. I'm beginning to learn machine learning. ...
74 views

### Momentum updates average of $g$, Adagrad also of $g^2$ - any other interesting updated averages for SGD convergence?

Updating exponential moving average is a basic tool of SGD methods, starting with of gradient $g$ in momentum method to extract local linear trend from the statistics. Then e.g. Adagrad, ADAM family ...
366 views

### Saddle-free Newton method for SGD - while Newton attracts saddles, is it worth to actively replel them?

While 2nd order methods have many advantages, e.g. natural gradient (e.g. in L-BFGS) attracts to close zero gradient point, which is usually saddle. Other try to pretend that our very non-convex ...
31 views

### When can I say a network converges better?

I trained two networks with and without skip connection. The network is a FCNN and has an encoding-decoding structure. I trained the networks with SGD and MSE for 150 epochs. The attached image is a ...
40 views

### How can I improve a classification algorithm for dogs and cats?

The following code is a ML algorithm trained to classify between dogs and cats, the database is composed by 25000 images (evenly split) and can be obtained at this Link (if you click it will ...
7 views

### Difference between using SGD on new data in batches vs training on the whole dataset (recommender systems)

I work with recommender systems and use SGD to train them. I am doing both: real-time updates: updating weights as soon as a new batch of 64 entries come in and training on the whole (well part of ...
150 views

7k views

### For convex problems, does gradient in Stochastic Gradient Descent (SGD) always point at the global extreme value?

Given a convex cost function, using SGD for optimization, we will have a gradient (vector) at a certain point during the optimization process. My question is, given the point on the convex, does the ...
279 views

### What is neural network good accuracy

I am very new at machine learning, and I'm building an artificial neural network that aims to classify inputs into 2 labels. I am training the network with randomly initialized weights and through the ...
86 views

### On projected gradient descent and inequality constraints

Consider the optimization problem $$\min_{x\in\mathbb{R}^n} \quad f(x)$$ using the gradient descent, we can iteratively solve this problem x^{k+1} = x^k-\...
55 views

Here's a toy dataset. ...
764 views

Consider the primal SVM problem: $$\frac12||w||^2 +\frac Cm \sum_{i=1}^m \max(0,1-y_iw\cdot x_i)$$ We want to find a solution with a bounded norm, by using SGD with a projection onto the convex set:...
178 views

### How to define nearest neighbor search such that it can be optimized using stochastic gradient descent?

Assume that there is a reference two-dimensional array ref and a given vector x. I would like to return the closest vector to <...
34k views

### How does batch size affect convergence of SGD and why?

I've seen similar conclusion from many discussions, that as the minibatch size gets larger the convergence of SGD actually gets harder/worse, for example this paper and this answer. Also I've heard of ...