New answers tagged

1 vote
Accepted

1D cluster - Jenks optimization - Finding optimal number

This seems to be a two stage problem: first, identify the number of clusters and then, secondly, optimally perform the clustering. For the first part, I'd suggest Cluster Validation by Prediction ...
user avatar
2 votes
Accepted

Should folds in k-fold CV actually be representative?

This is a reasonable question but it might be conflating 2 things. Should $k$ be chosen to make [something] representative? Should the-way-we-assign-data-to-each-fold be chosen to make [something] ...
user avatar
  • 1,439
1 vote

Why do we use Acquisition Functions?

The purpose of Bayesian optimization is to find global minima of functions that have many local minima. More typical optimizers are "local," in the sense that they follow some procedure ...
user avatar
  • 78.2k
2 votes

Equality-constrained least-squares when the matrix is singular

If it is possible to obtain a unique solution to $Ax=b$ subject to the constraints $Cx =d,$ then you can obtain it with Lagrange multipliers. To be explicit, suppose $A$ is an $n\times p$ matrix, $x$ ...
user avatar
  • 287k
1 vote

Finding the global minimum of a nonnegative, black-box, polynomial spline

It's not entirely clear to me what you do and don't know in your optimisation, but if I understand correctly, you don't even know how many piecewise parts there are to your function. If the number of ...
user avatar
  • 94.4k
1 vote
Accepted

Support vector machine, complementary slackness and marginal hyperplane

Theoretically, no. You can see this in terms of smoothly adjusting $C$ - at some point $x_i$ may be on the margin but with $\alpha_i = 0$. Practically, yes. In particular, note that stopping ...
user avatar
  • 1,259
0 votes

How to handle weighted examples in stochastic gradient descent (with mini-batches)?

Gradient descent is about following the derivatives (gradients). Recall that the derivative rule for calculating the derivative of a function $f$ times a constant $c$ is just $$ \frac{\partial}{\...
user avatar
  • 113k
1 vote
Accepted

How to perform fine balance matching in R for several covariates?

rcbalance() doesn't actually produce fine balance on the included covariates; it produces "refined" balance, which is slightly different in that it ...
user avatar
  • 21.8k
1 vote
Accepted

Why use transpose of nabla in gradient descent

Fundamentally this is just a notational issue. Assuming $x_k$ and $d_k$ are multidimensional "vectors" (say $n$ dimensional), the standard is often that all vectors are "columns", ...
user avatar
0 votes

Why use transpose of nabla in gradient descent

Note that if f is scalar, you need an inner product to make a scalar on the right hand side. Generally nabla will be a vector
user avatar
  • 5,176
0 votes

Simulated Annealing Parameter Tuning

Just looking through the code: SA function is always called with the same value of temperature, $T=8$. The statement ...
user avatar
  • 1,580
2 votes
Accepted

Combinations from different sets with weightings

This can be expressed as a binary linear program. Provided the total number of unique combinations is not prohibitively large, it can be solved efficiently. When multiple solutions exist, usually ...
user avatar
  • 287k
2 votes

Why do we use Acquisition Functions?

In uncertain scenarios, where Bayesian optimization is employed, we are facing the exploration-exploitation tradeoff. The acquisition function is a solution for it. As you noticed, Bayesian ...
user avatar
  • 113k
0 votes

Examples in the Real World where Evolutionary Algorithms/Genetic Algorithms Outperform other Classes of Optimization Algorithms

It's still an open field of research, but there are many examples. Hyperparameter tuning is an example of a noisy and expensive function that can be handled by evolutionary algorithms. Neural ...
user avatar
  • 113k
0 votes

What does it mean when all gradients of a neural network are 0?

Gradients all equal to zero does not necessarily imply any problem with the network. Both minima and maxima occur where the gradient is zero. So it’s possible that your network has arrived at a local ...
user avatar
  • 78.2k
-2 votes

What does it mean when all gradients of a neural network are 0?

If you're using "tensorflow" and giving your model a batch make sure you put model(batch, training=True). If training isn't set to true then the gradient ...
user avatar

Top 50 recent answers are included