Vector pointing in the direction where a function is growing fastest; its components are partial derivatives of this function. For questions about gradients in ecology, please use the [ecology] tag instead.

130 questions
Filter by
Sorted by
Tagged with
22 views

130 views

### Is there an advantage to normalizing labels when using MSE loss?

I am designing a NN that uses MSE as a loss regressor. Its a big network and when I train, the loss/gradients are HUGE. I have to clip my gradients our else the loss just goes to NaNs. The differences ...
34 views

### In Gradient Boosting Tree, why do we fit the tree on the residuals and not on the sum of the previous function and the residuals?

In the Gradient Boosting Tree algorithm, as described in https://en.wikipedia.org/wiki/Gradient_boosting#Gradient_tree_boosting, we update the previous model $F_m$ by adding the results $h_m$ of the ...
1k views

### Gradient Boosting - Price Forecast based on time series data [closed]

What I am trying to achieve. I want to forecast Natural Gas prices under the column "NG Open" based on other parameters in the data set below for all Contract Months ,which is scraped from a public ...
31 views

### Storage and re-computation of Intermediate / Weight / Back-propagated Gradients

I need to track the computation and storage of different parts of my network training. To be on the same page, let's assume the simple following scenario (biases omitted) Questions Local Gradients - ...
175 views

### Gradient Descent in Metric Learning for Kernel Regression (MLKR)

I am currently studying the Metric Learning for Kernel Regression (MLKR) algorithm (http://proceedings.mlr.press/v2/weinberger07a/weinberger07a.pdf). Let $\{(x_{1}, y_{1}), ..., (x_{N}, y_{N})\}$ ...
157 views

### focusing on hard examples in neural networks, like in gradient boosting?

gradient boosting can be seen as focusing on the hard examples (the training set examples where the prediction is still far from the true label, and the gradient is still big). is there a similar ...
1k views

440 views

### How do the residual blocks prevent exploding gradients?

I am reading Roger Grosse's lecture notes on ResNet and I have a question regarding the explanation on how residue blocks prevent gradient explosion, see the screenshot below: My confusion is: this ...
237 views

### How do I implement softmax forward propagation and backpropagation to replace sigmoid in a neural network?

I'm currently using 3Blue1Brown's tutorial series on neural networks and lack extensive calculus knowledge/experience. I'm using the following equations to calculate the gradients for weights and ...
127 views

This is the value function from the GANs paper: The authors explain that this equation "may not provide sufficient gradient for $G$ to learn well", because early in the learning process the ...
54 views

### Can a full batch gradient descent point not to a minimum for a convex function?

Let's say that we have a three dimentional convex function with a minimum marked by a red dot. Can a full batch gradient in a blue point not directed to a red dot as drawn below? Actually this ...
436 views

### Gradient ascent to maximise log likelihood

I'm working on an online method to adapt the parameters $\mu, \Sigma$ of a Gaussian distribution. Do to so i perform a gradient descent on the log likelihood $L$. With the help of the matrix cookbook ...
This question was asked about the gradient of a linear function. In the answer I don't understand why is $\nabla g(w)=X^t$ (the marked in red part)? Shouldn't it be just $\nabla g(w)=X$? And in ...