Tag Info

Accepted

Reconciling optimisation for log-likelihood and Brier score

Although both log-loss and Brier scores provide proper scoring rules, they put emphasis on different regions of the probability distributions. Quoting from Wikipedia: the choice of a scoring rule ...
• 96.3k
Accepted

Why go through the trouble of expectation maximization and not use gradient descent?

There are several advantages of the EM algorithm over gradient descent: Monotonic convergence. The EM algorithm never decreases the log-likelihood. This is not necessarily true for gradient descent. ...
• 21.4k

A Simple Toy ML problem that surprisingly fails to converge (or even "try"!)

The problem isn't learnable because of how the data are generated. Without loss of generality, we can consider vectors with 2 elements. Then the values of each vector are given by a single scalar, an ...
• 92.3k

Example where the initial random state of a logistic regression matters?

Both linear and logistic regression are convex optimisation problems and have same behaviour. If the 2nd derivative of the objective at the minimum is positive definite, then the minimum is unique, ...
• 7,238
Accepted

Censored Regression model

After replacing pnorm(z) with pnorm(xb/sigma), I find: ...
• 19.5k

• 1,424
Accepted

Closed Form Solution for Gaussian Matrix which is Convex Combination?

The answer to your question Given $\lambda$, we can calculate $\mu_x, \mu_y, \sigma_x, \sigma_y$ using standard techniques, which you can find by searching "Weighted Least Squares" and ...
• 4,600

Why do we make a big fuss about using Fisher scoring when we fit a GLM?

A good discussion of GLM fitting algorithms, including a comparison with Newton-Raphson (which uses the observed Hessian as opposed to the expected Hessian in the IRLS / Fisher scoring algorithm) and ...
• 3,299

what is the mistake of convergence proof in Adam

I believe this is covered in David Martínez-Rubio's MSc thesis https://damaru2.github.io/convergence_analysis_hypergradient_descent/dissertation_hypergradients.pdf see pages 18 & 19 of thesis, ...
• 7,238

Why can the method of moments be expressed as a minimization problem?

To expand on @Glen_b’s comment, consider the function $f : \mathbb R \to \mathbb R$, and suppose that we want to find the set of all $x \in \mathbb R$ such that $f(x) = 0$. It may be the case that ...
• 5,100
Accepted

You are confusing the current value of the parameters at which the gradient is calculated with the data points. Consider the simple model: $$y_i = a + bx_i + u_i$$ and let us choose to minimize the ...
• 40.5k

Why go through the trouble of expectation maximization and not use gradient descent?

Expectation-Maximization can be seen as a form of gradient descent which has been specifically tailored for latent variables models. In a latent variables model, you actually have 2 sets of unknowns: ...
• 2,696
Accepted

Why does nlme::fdHess return NaNs when evaluated at 0?

Looking at the code it seems as though the basic problem here is that the function, by default, chooses the finite difference step size relative to the absolute value of the parameters: in the ...
• 44.6k
Accepted

Difference between Bayesian optimization and multi-armed bandit optimization

Bayesian optimization can be considered as an infinite-armed bandit algorithm. My understanding for why we don't use the same term for both is the scope of their applications and different subfields ...
• 46