The tag has no usage guidance.

206 questions
Filter by
Sorted by
Tagged with
26 views

261 views

### Is this "stochastic gradient method" different from the stochastic gradient descent algorithm?

In a computational statistics book, I found an optimization method to find local minimum of a function. Let's assume that we have a differentiable function $f: \mathbb{R}^2 \longrightarrow \mathbb{R}$....
1 vote
17 views

### Why training is smoother when no randomization of batch after each epochs is applied?

I am wondering why training error converges more smoothly when I am not applying randomization after each epochs? After each epoch I am taking the loss obtained in my last minibatch. I see that when ...
14 views

### Loss is growing between epochs in simple model [duplicate]

I am new to ML and trying to write a very simple model that can predict the future of a function given the past. That is to say, I define a simple cubic function (y = x^3 + x^2 + x) and apply it to a ...
1 vote
61 views

### Why do we need stochastic gradient descent when we can precompute certain results in batch gradient descent?

I have read that batch gradient descent forces this summation at every step of the update, which makes it time consuming. But if we have the following hypothesis function: $$h(x^i) = w_0 + w_1x^i$$ ...
146 views

### VGG-16 Why Validation loss is going up and down?

I am implementing VGG-16 using this cats vs dogs dataset, and am comparing the results with SGD vs Adam and getting similar results. Here's my parameters for using Adam ...
297 views

### Which methods can improve model fit through gradient descent?

I am looking to solve the following question: Which methods below can improve model fit (MSE) through gradient descent? Use a high learning rate Use stochastic gradient descent Start with a high ...
1 vote
101 views

I'm trying to understand the differences between the update rule for stochastic gradient ascent and descent. I've read some articles and still don't understand how to calculate the update rule: ...
656 views

1 vote
95 views

### Which is better: performance stability or high performance in the initial epochs but extremely unstable performance during validation?

I am working on a multiclassification problem using time series data. I am using a hybrid model (such as LSTM, CNN, attention, etc.). I tried two optimizers, ADAM with 0.001 learning rate and SGD with ...
1 vote
81 views

### details of stochastic Expectation Maximization (EM) algorithm

I went into a paper, Online EM Algorithm for Latent Data Models (Olivier Cappé & Eric Moulines, 2009). I got confused by the first equation the authors wrote, the Q function: Here the authors ...
31 views

### What could go wrong if I do gradient descent by class?

For example, when training on CIFAR10, each minibatch typically contains images from all 10 classes (assuming a moderately large batch size such as 64). What could go wrong if I train on a homogeneous ...
617 views

### How to find the gradient when a black box I/O function is involved in evaluation of the loss?

I am trying to learn a neural network $NN_\pi$ to minimize the loss function $$L_{\pi} = || Y_{true} - F(X_{true}, NN_{\pi}(X_{true}) ) ||^2$$ where $F$ is a black box (I/O) function (we only ...
352 views

### How to handle weighted examples in stochastic gradient descent (with mini-batches)?

Suppose I have $M$ data points $x_i$ and associated weights $w_i > 0$. I want to optimize a function, $$F(\theta) = \frac{1}{M}\sum_i w_i f(x_i;\theta)$$ in the parameters $\theta$. I will assume ...
1 vote
108 views

### Regular *negative* spikes in neural network training loss

I am training an ALL-CNN network using the Adam solver. As the figure shows, the testing seems to converge to an acceptable solution, but there are these regular negative spikes during training that ...
175 views

### Mini-Batch Gradient Descent - Large batch size require small learning rate?

Coursera Machine Learning in the Enterprise - Science of Machine Learning and Custom Training says large batch size require smaller LR. However, How should the learning rate change as the batch size ...
1 vote
183 views

### Derive the gradient matrices w.r.t. W1 and W2 and backprop equation in a Residual Network [closed]

How would I go about deriving gradient matrices w.r.t. W1 and W2 and backpropagation equation in a residual block that is a part of a larger ResNet network with forward propagation expressed as:  F(...
1 vote
120 views

### How does SGD training error decrease in subsequent epochs with non-iid samples when it is recommended that samples in subsequent epochs be iid?

I have been reading the Deep Learning book by Ian Goodfellow and on pg. 277, they mention: It is also crucial that the minibatches be selected randomly. Computing an unbiased estimate of the expected ...
320 views

### which training mode is more convenient for small datasets?

I have a regression problem to be solved using one of neural networks models, but I have a small dataset which contains 30 samples. Which training mode is more suitable for such dataset: stochastic or ...
153 views

### Bias introduced when using weak shuffling

I have batch learning problem (in this particular case a neural network) where I am training my data in batches, and then repeating for a number of epochs. In Stochastic Gradient Descent, we minimise ...
115 views

### About update procedure in data incremental learning

As far as I understood, the idea of data incremental learning consists of keeping the model always up to date. Suppose that we trained a model for user recognition using voice as input. Therefore, the ...
840 views

### Why does ADAM optimization perform well on non-convex functions and bad on convex functions?

I'm currently trying to understand SGD and ADAM optimization, and I understand that ADAM optimization performs well on non-convex loss functions and that SGD performs well on convex loss functions (...
1 vote
228 views

### Does always gradients in mini-batch SGD have to be unbiased in order to prove convergence?

I am currently reading this paper  and . The authors state that: Our analytical results include almost all of the unbiased compression techniques. And also: (i) gradient compression must be ...
119 views

### Stochastic Gradient Descent Code Check for Least Squares

I have an R-based implementation of the gradient descent and am trying to also get it to work as SGD. The function matches R's lm function when using the entire data set. But, when I sample from the ...
1 vote
185 views

### Performing Linear Regression using Stochastic Gradient Descent, by batches

I am presented with a data set, where I am supposed to perform linear regression on this using SGD. My first instinct would be to train each data point there is until I reach the last one. Only then ...
675 views

### Why is a 2nd order derivative optimization better for no hidden layer neural networks?

I was reading in this blog. That first order derivative SGD optimization methods are worse for neural networks without hidden layers and 2nd order is better, because that's what regression uses. Why ...
361 views

### Why does stochastic gradient descent lead us to a minimum at all?

Why do we think that stochastic gradient descent is going to find a minimum at all? I mean on each iteration SGD moves in the direction that reduces only current batch's error (SGD doesn't care about ...
514 views

### The reason (and intuition) behind why stochastic gradient descent can get stuck on a local minimum

Suppose you want to find $k$ that minimises your cost function $J(k)$. We may want to apply batch gradient descent or stochastic gradient descent. Let's deliberately initialise $k$ with the same ...
24 views

### Can stochastic gradient descent for Bayesian Inference? [duplicate]

I was looking at the Bayesian MAP estimate formula which is the "argmax(likelihood * prior)". Can this be calculated using stochastic gradient descent? Gradient descent requires knowing the ...
1 vote
124 views

### Is outer product of marginal distribution the "best" mean-field approximation for a joint distribution?

I am certain this has been asked somewhere else, if that's the case, link me and close the thread. I am studying variational inference and mean-field approximation. All the explanations I come across ...
34 views

### Why a minimiser of a subset of training dataset is that of the whole training set

In section 3.3 of Bottou et al (2018), under the 'intuitive motivation' paragraph, the authors claim that 'a minimiser of empirical risk for the larger set $S$ is clearly given by a minimiser for the ...
635 views

### upsampling vs class weights in mini-batch SGD

Let's consider using mini-batch SGD in (neural network) binary classification problem with imbalanced dataset. Let's say that the ratio between the number of examples in each class is positive:...
1 vote
2k views

### When are very small learning rates useful?

I just wondered if there are cases where small or very small learning rates in gradient descent based optimization are useful? A large learning rate allows the model to explore a much larger portion ...
54 views

### Gradient Descent Algorithm for Interdependent parameters

Suppose I have $n$ data points ($X_i$,$y_i$) where $X_i$ is a vector and $y_i$ is a scalar, $1 \le i \le n$. By defining $\hat{\boldsymbol{Y}} = \boldsymbol{\Theta} \boldsymbol{X} + \boldsymbol{b}$ ...
5k views

### Loss values above 1.0

I have a convolutional neural network for tensors classification in Pytorch. I am using Cross-Entropy Loss. My optimizer is Stochastic Gradient Descent and the learning rate is 0.0001. The accuracy of ...