We’re rewarding the question askers & reputations are being recalculated! Read more.

Questions tagged [gradient]

Vector pointing in the direction where a function is growing fastest; its components are partial derivatives of this function. For questions about gradients in ecology, please use the [ecology] tag instead.

Filter by
Sorted by
Tagged with
1
vote
0answers
318 views

Is there Early Stop in Stochastic gradient descent?

Let's say I have 100 data points. If stochastic gradient didn't converge at first round of 100 data points, then I need to continue another round. Then if it converged at 50th data point, so total ...
3
votes
1answer
942 views

Standardizing numerical and encoding of categorical data for training boosted decision tree

Is there a "best practice" way of standardizing numerical and encoding of categorical data for training boosted decision tree? Both for classification and regression problems
12
votes
1answer
8k views

Is gradient boosting appropriate for data with low event rates like 1%?

I am trying gradient boosting on a dataset with event rate about 1% using Enterprise miner, but it is failing to produce any output. My question is, since it a decision tree based approach, is it even ...
0
votes
0answers
749 views

Why does the basic gradient descent not converge for this example?

I have a toy example for a linear regression of the form $$y=\beta_0 + \beta_1x_1 + \beta_2x_2$$ The data is: ...
9
votes
2answers
4k views

Deriving gradient of a single layer neural network w.r.t its inputs, what is the operator in the chain rule?

Problem is: Derive the gradient with respect to the input layer for a a single hidden layer neural network using sigmoid for input -> hidden, softmax for hidden -> output, with a cross entropy ...
1
vote
0answers
1k views

Deriving the gradient of a loss function for generalized logistic regression

I am trying, without much success so far, to derive the gradient of the following cost function in order to fit a logistic curve to some data: $J(a, k, b, m) = \sum_i^n(y_i - a + \frac{k - a}{(1 + e^{...
3
votes
2answers
3k views

Numerical check of gradient in neural network

I am trying to check if my implementation of backpropogation is correct by checking the calculated gradients with the numeric gradient. I am testing it on a very simple linear network (i.e. no ...
3
votes
1answer
432 views

How to compute gradient of partial log-likelihood function in Cox proportional hazards model?

The partial log-likelihood function in Cox proportional hazards is given with such formula $${}_{p}\ell(\beta) = \sum\limits_{i=1}^{K}X_i'\beta - \sum\limits_{i=1}^{K}\log\Big(\sum\limits_{l\in \...
35
votes
2answers
13k views

Gradient Boosting for Linear Regression - why does it not work?

While learning about Gradient Boosting, I haven't heard about any constraints regarding the properties of a "weak classifier" that the method uses to build and ensemble model. However, I could not ...
8
votes
3answers
5k views

Numeric Gradient Checking: How close is close enough?

I made a convolutional neural network and I wanted to check that my gradients are being calculated correctly using numeric gradient checking. The question is, how close is close enough? My checking ...
7
votes
2answers
17k views

How to use XGboost.cv with hyperparameters optimization?

I want to optimize hyperparameters of XGboost using crossvalidation. However, it is not clear how to obtain the model from xgb.cv. For instance I call ...
1
vote
2answers
3k views

Calculating gradient of a function for optimization

I need to optimize a function. This function is a likelihood function which takes a set of parameters (to be optimized) and calculates the likelihood (to be optimized) as a result. ...
1
vote
1answer
55 views

RVM gradient of likelihood

Can somebody tell me how Tipping (2001) in his classical paper about the Relevance Vector Machine arrives at the following expressions \begin{align} \mathbf{D} &= (\mathbf{C}^{-1}\mathbf{t}\mathbf{...
1
vote
0answers
68 views

Gradient of a sum of indicators

EDITED w.r.t. whuber's comment: Say I have a function $\mathbb R^n \rightarrow \mathbb R$: $$f(w_1,\ldots,w_n) = \frac{n^-\sum_{i\in I^-}w_ix_i}{n^+\sum_{i\in I^+}w_ix_i}$$ with fixed $x_i\in\mathbb ...
0
votes
1answer
328 views

Defining grad in R's optim for MLE

I have a ML I want to maximize in R's function optim. I am currently using the method BFGS. The optim procedure is quite slow however, and I was hoping to speed up the process by specifying the ...
3
votes
1answer
10k views

Gradient of loss function for (non)-linear prediction functions

$ \newcommand{\y}{\mathbf{y}} \newcommand{\wv}{\mathbf{w}} \newcommand{\xv}{\mathbf{x}} \newcommand{\loss}{L(\wv;\xv, y)} $ I'm trying to clear up the calculation of the gradient of a loss function, ...
2
votes
0answers
69 views

Intuition about the gradient of the constraint

I was rethinking the logic of solving constraint optimization problem. And I read many stuff in website like this and textbooks. But the following question is still unsolved (to me). We know that at ...
2
votes
1answer
1k views

Statistically testing for a significant difference between two slope values [closed]

I have five trend lines plotted in excel of number of prescriptions of a 5 different drugs over time (MM/YYYY) and I want to test the statistical significance of the difference between the slopes, to ...
1
vote
3answers
3k views

Implementing WARP Loss (Gradient Computation)

I am trying to implement the WARP Loss in Torch, as defined in the WSABIE paper: http://www.thespermwhale.com/jaseweston/papers/wsabie-ijcai.pdf The Algorithm is as follows: The Algorithm specifies ...
4
votes
1answer
920 views

Temporal convolution for NLP [closed]

I'm trying to follow Kalchbrenner et al. 2014 (http://nal.co/papers/Kalchbrenner_DCNN_ACL14) (and basically most of the papers in the last 2 years which applied CNNs to NLP tasks) and implement the ...
2
votes
1answer
834 views

Is the gradient computation in the word2vec implementation actually wrong?

In the paper "Efficient Estimation of Word Representations in Vector Space", it is stated that "All models are trained using stochastic gradient descent and backpropagation": http://arxiv.org/pdf/1301....
0
votes
0answers
611 views

Convert time scalar field to velocity vector field

I am trying to figure out if it's possible to create vector velocity field from some raster containing spatial distribution of scalar variable (timestamp of some mass-media event in my case). Here is ...
7
votes
2answers
3k views

Name for outer product of gradient approximation of Hessian

Is there a name for approximating the Hessian as the outer product of the gradient with itself? If one is approximating the Hessian of the log-loss, then the outer product of the gradient with itself ...
1
vote
1answer
126 views

Distribution of log-likelihood gradient

My question is simple: Is there any results regarding the distribution of log-likelihood function gradient? It may be asymptotic results as well.