36

What am I missing here? I don't think you're really missing anything! Another observation is that a sum of subsequent linear regression models can be represented as a single regression model as well (adding all intercepts and corresponding coefficients) so I cannot imagine how that could ever improve the model. The last observation is that a linear ...


17

This is how I have trained a xgboost classifier with a 5-fold cross-validation to optimize the F1 score using randomized search for hyperparameter optimization. Note that X and y here should be pandas dataframes. from scipy import stats from xgboost import XGBClassifier from sklearn.model_selection import RandomizedSearchCV, KFold from sklearn.metrics ...


12

The derivatives are with respect to $x$ (or y_hat in the code) instead of $p$. As you've already derived (Edit: as Simon.H mentioned, since the actual loss should be the negative log likelihood, so I've changed the sign of your result) $$\frac{\partial f}{\partial p}=\frac{p-y}{\left(1-p\right)p},$$ and the derivative of sigmoid is $$\frac{\partial p}{\...


12

If Gradient Descent gets initialized in such a way that it starts at a local maximum (or a saddle point, or a local minimum) with gradient zero, then it will simply stay stuck there. Variations of GD, such as Stochastic GD and Mini-batch GD try to work around this by adding an element of randomness to the search, but even those aren't guaranteed to escape a ...


9

The closest I have seen to addressing this was in the Stanford UFLDL tutorial within the softmax regression section. Copying the key statement: The norm of the difference between the numerical gradient and your analytical gradient should be small, on the order of $10^{-9}$. In python the code would look something like this: norm(gradients - ...


9

If you do not carefully choose the range of the initial values for the weights, and if you do not control the range of the values of the weights during training, vanishing gradient would occur which is the main barrier to learning deep networks. The neural networks are trained using the gradient descent algorithm: $$w^{new} := w^{old} - \eta \frac{\partial L}...


7

(To give short answer to this:) It is fine to use a gradient boosting machine algorithm when dealing with an imbalanced dataset. When dealing with a strongly imbalanced dataset it much more relevant to question the suitability of the metric used. We should potentially avoid metrics, like Accuracy or Recall, that are based on arbitrary thresholds, and opt ...


7

Background Theory that's helpful One small fact that you can use to help understand whether a numeric derivative is correctly calculated or not is the Cauchy Remainder of the Taylor expansion. That is, $f(x + h) = f(x) + hf'(x) + \frac{h^2}{2}f''(\xi)$ for some $\xi \in [x, x+ h]$ This is helpful, because you've probably approximated your first derivative ...


7

This problem is sometimes called score estimation, because $\nabla_x \log p(x)$ is the score of whatever model $p$ with respect to a hypothetical location parameter. I've seen it called the Hyvärinen score after Aapo Hyvärinen's technique called score matching for fitting (unnormalized) statistical models. Some flavors of score estimation: From a density ...


6

The expected value of the outer product of the gradient of the log-likelihood is the "information matrix", or "Fisher information" irrespective of whether we use it instead of the negative of the Hessian or not, see this post. It is also the "variance of the score". The relation that permits us to use the outer product of the gradient instead of the ...


6

It's hard to know for sure with such a terse and pithy description, but here's a shot at what he may likely be getting at. Say you have a very high cardinality feature $x$ with some giant set of possible levels $l_1, l_2, \cdots, l_n$. These can be difficult to use in a model directly. One approach to deriving a feature from such a predictor is to take ...


6

A regression tree makes sense. You 'classify' your data into one of a finite number of values. Note, that while called a regression, a regression tree is a nonlinear model. Once you believe that, the idea of using a random forest instead of a single tree makes sense. One just averages the values of all the regression trees. Once one has a regression ...


6

Here is my try $$J(x) = -\frac{1}{m}\sum_{i = 1}^{m} b_iln(h_i) + (1 - b_i)ln(1 - h_i)$$ where $h_i = \sigma(x^Ta_i)$. Let $A = [a_1^T, \dots, a_m^T]^T$. Assuming $ln, \sigma, \frac{1}{\cdot}$ work element-wise on vectors, $\odot$ is element-wise multiplication and $\mathbb{1}$ is a vector of $1$s we have $$J(x) = -\frac{1}{m}\big[b^Tln(\sigma(Ax)) + (\...


6

The Mean Absolute Percentage Error (MAPE) is defined as $$\text{MAPE} := \frac{1}{N}\sum_{i=1}^N\frac{|\hat{y}_i-y_i|}{y_i},$$ where the $y_i$ are actuals and the $\hat{y}_i$ are predictions. The gradient is the vector collecting the first derivatives: $$\frac{\partial\text{MAPE}}{\partial\hat{y}_i} = \begin{cases} -\frac{1}{N}, & \text{ if } \hat{y}...


6

For a typical loss function $L = E_{x_i \sim \text{D}}[f(x_i)]$ and true gradient $\nabla L = E[\nabla f(x_i)]$, the expectation of the SGD gradient is $E[\nabla f(x')]$ where $x'$ is the datapoint in our batch of size this is 1. This is clearly unbiased. The loss function in the paper takes the form $L = \log E[e^{f(x)}]$ and has the gradient $$\nabla L = \...


5

Please refer to this tutorial http://cs231n.github.io/neural-networks-3/#ensemble. The "Gradient Check" section is very detailed and helpful. As is suggested by gung, I include the main points of this link: Use $\frac{f(w+h)-f(w-h)}{2h}$ approximation, where $h\sim 10^{-5}$. Monitor the fraction of $\frac{|f_a'(w)-f_n'(w)|}{max(|f_a'(w)|,|f_n'(w)|)}$, ...


5

The chain rule is $$ (f(g(x)))' = f'(g(x))g'(x). $$ In a multivariate setting, with $f:\mathbb{R}\rightarrow\mathbb{R}$ and $g:\mathbb{R}^p\rightarrow\mathbb{R}$ this becomes $$ \nabla f(g(w)) = \nabla g(w)f'(g(w)). $$ Remember that the chain rule is $$ f(g(x)) = \frac{\partial z}{\partial x}\frac{\partial f}{\partial z}, $$ where $z = g(x)$. ...


5

The least squares projection matrix is given by $X(X^{T}X)^{-1}X^{T}$ We can use this to directly obtain our predicted values $\hat{y}$, e.g. $\hat{y} = X(X^{T}X)^{-1}X^{T}y $ Let's say you fit a regression and subsequently you calculate your residuals $e = y - \hat{y} = y - X(X^{T}X)^{-1}X^{T}y $ And then you use this residual vector e as your new ...


5

As noted in the comments, it is best to write out the matrix equations and then apply the standard derivative rules. After a bit of experience with small cases, where you expand out all the terms, you can try just writing out the equations using summations and subscripts. In terms of mathematical rules, for completeness I will note that there is a pretty ...


5

After I obtained some help from the authors, I can write down now how I understand it. Somebody jump in, if there is disagreement. Say, we have some differentiable loss function $L(y,H(x))$ , where $H(x)$ is our tree ensemble at some iteration. Let $g_i$ be the gradient of our loss function at some entry corresponding to observation i. In each iteration, ...


4

For the probabilities that "look the same", either there is not enough information in your data to better distinguish the two classes, and thus the prediction is mostly based on the number of instances rather than on the features informations, or you do not give enough power to XGBoost to fit the dataset. From your question, I think the latter is more likely....


4

I believe that the key to answering this question is to point out that the element-wise multiplication is actually shorthand and therefore when you derive the equations you never actually use it. The actual operation is not an element-wise multiplication but instead a standard matrix multiplication of a gradient with a Jacobian, always. In the case of the ...


4

The outer product of gradient estimator for the covariance matrix of maximum likelihood estimates is also known as the BHHH estimator, because it was proposed by Berndt, Hall, Hall and Hausman in this paper: Berndt, E.K., Hall, B.H., Hall, R.E. and Hausman, J.A. (1974). "Estimation and Inference in Nonlinear Structural Models". Annals of Economic and ...


4

$$ E_Q\left[\frac{\nabla_\phi Q_\phi(h|x)}{Q_\phi(h|x)}\right] = \int\frac{\nabla_\phi Q_\phi(h|x)}{Q_\phi(h|x)} Q_\phi(h|x) = \int{\nabla_\phi Q_\phi(h|x)}$$ Assuming that you can exchange the integral and the gradient operators (deep waters) $$ = \nabla_\phi\int{ Q_\phi(h|x)} = \nabla_\phi E_Q[1] = \nabla_\phi 1 =0$$ Since the distribution $Q$ is ...


4

There is something "wrong" with the algorithm, gradient descent (its very name is an example of false advertising). That doesn't mean there is a mistake in your implementation. Gradient descent, i.e., steepest descent without use of line search or trust regions, need not converge, and can actually diverge, even on a strictly convex function, even in one ...


4

Use the infinity norm (i.e., maximum absolute deviation across all components), not the 2 norm. Then it will essentially be independent of number of components (dimension). If you do gradient check on at least some components, that's better than nothing. And you do it once, when you first run your model, not every time you run it. You know why gradient ...


4

Continuing from comments, when you use sigmoid activation function which squashes the input to a small range $(0,1)$, you further multiply it by a small learning rate and more partial derivatives (chain rule) as you go back in layers. The value of delta to be updated diminishes and thus earlier layers get little or no updates. If little, then it would ...


4

Consider the following feedforward neural network: Let $w^l_{j,k}$ be the weight for the connection from the $k^{\text{th}}$ neuron in the $(l-1)^{\text{th}}$ layer to the $j^{\text{th}}$ neuron in the $l^{\text{th}}$ layer. Let $b^l_j$ be the bias of the $j^{\text{th}}$ neuron in the $l^{\text{th}}$ layer. Let $C$ be the cost function. We consider the ...


4

One way to look at this is by Taylor approximation. Remember $$f(x+\Delta x)\approx f(x)+\Delta x f'(x)+\frac 1 2 \Delta x^2 f''(x)+\frac 1 6 \Delta x^3f'''(x)+\dots$$ One sided looks like this $$\frac{f(x+\Delta x)-f(x)}{\Delta x}\approx f'(x)+\frac 1 2 \Delta x f''(x)$$ Two sided looks like this $$\frac{f(x+\Delta x)-f(x-\Delta x)}{2\Delta x}\approx f'...


4

In short answer, the gradient here refers to the gradient of loss function, and it is the target value for each new tree to predict. Suppose you have a true value $y$ and a predicted value $\hat{y}$. The predicted value is constructed from some existing trees. Then you are trying to construct the next tree which gives a prediction $z$. Then your final ...


Only top voted, non community-wiki answers of a minimum length are eligible