We’re rewarding the question askers & reputations are being recalculated! Read more.

Vector pointing in the direction where a function is growing fastest; its components are partial derivatives of this function. For questions about gradients in ecology, please use the [ecology] tag instead.

122 questions
Filter by
Sorted by
Tagged with
37 views

### Why do we use the log-derivative trick before Monte Carlo?

I still don't understand how we can approximate the gradient of an expected value... Indeed it's impossible to sample points and then to average the gradients of them as we have only samples... (How ...
7 views

### Gradient for log regression loss [migrated]

I'm trying to write mini-batch gradient descent for log regression. $\nabla L = - \sum_{i=1}^{m} (y_i - \sigma(\left<w,x_i\right>))\:x_i$ Given numpy matrices ...
41 views

### What is state of the art in gradient free neural network learning, esp. for images?

I've been recently looking into gradient free learning of neural networks. However, most of the techniques I've found seem to be only applied to toy problems, which I assume means they're infeasible ...
10 views

### Whats a good estimation for error measuremets when trying to predict values inside two bands?

I am using gradient boosting to predict two quantiles (upper and lower). The predicted value can be above, below, or in bounds. The problem I am facing is that counting the number of values in bound ...
27 views

### Standard errors for Composite Marginal Likelihood

I am estimating a multivariate ordered probit model using a composite marginal likelihood (CML) approach. In other words, I replace the full likelihood function by a surrogate likelihood constructed ...
17k views

### How to use XGboost.cv with hyperparameters optimization?

I want to optimize hyperparameters of XGboost using crossvalidation. However, it is not clear how to obtain the model from xgb.cv. For instance I call ...
83 views

### Why is the expected gradient of a density not parallel to the expected gradient of the log density?

I'm confused by a seemingly counter-intuitive property of the interaction between distributions, log transforms, expectations and gradients. Suppose I have some distribution over random variable $x$ ...
30 views

5k views

### How to compute the gradient and hessian of logarithmic loss? (question is based on a numpy example script from xgboost's github repository)

I would like to understand how the gradient and hessian of the logloss function are computed in an xgboost sample script. I've simplified the function to take numpy arrays, and generated ...
1k views

### Can I combine many gradient boosting trees using bagging technique

Based on Gradient Boosting Tree vs Random Forest . GBDT and RF using different strategy to tackle bias and variance. My question is that can I resample dataset (with replacement) to train multiple ...
28 views

### gradient vs derivative: defintions of [closed]

According to wikipedia: In mathematics, the gradient is a multi-variable generalization of the derivative. Like the derivative, the gradient represents the slope of the tangent of the graph of ...
51 views

97 views

### Is there an advantage to normalizing labels when using MSE loss?

I am designing a NN that uses MSE as a loss regressor. Its a big network and when I train, the loss/gradients are HUGE. I have to clip my gradients our else the loss just goes to NaNs. The differences ...
29 views

### Storage and re-computation of Intermediate / Weight / Back-propagated Gradients

I need to track the computation and storage of different parts of my network training. To be on the same page, let's assume the simple following scenario (biases omitted) Questions Local Gradients - ...
33 views

### In Gradient Boosting Tree, why do we fit the tree on the residuals and not on the sum of the previous function and the residuals?

In the Gradient Boosting Tree algorithm, as described in https://en.wikipedia.org/wiki/Gradient_boosting#Gradient_tree_boosting, we update the previous model $F_m$ by adding the results $h_m$ of the ...
152 views

### Gradient Descent in Metric Learning for Kernel Regression (MLKR)

I am currently studying the Metric Learning for Kernel Regression (MLKR) algorithm (http://proceedings.mlr.press/v2/weinberger07a/weinberger07a.pdf). Let $\{(x_{1}, y_{1}), ..., (x_{N}, y_{N})\}$ ...
339 views