New answers tagged optimization
1
vote
Accepted
1D cluster - Jenks optimization - Finding optimal number
This seems to be a two stage problem: first, identify the number of clusters and then, secondly, optimally perform the clustering.
For the first part, I'd suggest Cluster Validation by Prediction ...
- 4,599
2
votes
Accepted
Should folds in k-fold CV actually be representative?
This is a reasonable question but it might be conflating 2 things.
Should $k$ be chosen to make [something] representative?
Should the-way-we-assign-data-to-each-fold be chosen to make [something] ...
- 1,439
1
vote
Why do we use Acquisition Functions?
The purpose of Bayesian optimization is to find global minima of functions that have many local minima. More typical optimizers are "local," in the sense that they follow some procedure ...
- 78.2k
2
votes
Equality-constrained least-squares when the matrix is singular
If it is possible to obtain a unique solution to $Ax=b$ subject to the constraints $Cx =d,$ then you can obtain it with Lagrange multipliers.
To be explicit, suppose $A$ is an $n\times p$ matrix, $x$ ...
- 287k
1
vote
Finding the global minimum of a nonnegative, black-box, polynomial spline
It's not entirely clear to me what you do and don't know in your optimisation, but if I understand correctly, you don't even know how many piecewise parts there are to your function. If the number of ...
- 94.4k
1
vote
Accepted
Support vector machine, complementary slackness and marginal hyperplane
Theoretically, no. You can see this in terms of smoothly adjusting $C$ - at some point $x_i$ may be on the margin but with $\alpha_i = 0$. Practically, yes. In particular, note that stopping ...
- 1,259
0
votes
How to handle weighted examples in stochastic gradient descent (with mini-batches)?
Gradient descent is about following the derivatives (gradients). Recall that the derivative rule for calculating the derivative of a function $f$ times a constant $c$ is just
$$
\frac{\partial}{\...
Tim♦
- 113k
1
vote
Accepted
How to perform fine balance matching in R for several covariates?
rcbalance() doesn't actually produce fine balance on the included covariates; it produces "refined" balance, which is slightly different in that it ...
- 21.8k
1
vote
Accepted
Why use transpose of nabla in gradient descent
Fundamentally this is just a notational issue. Assuming $x_k$ and $d_k$ are multidimensional "vectors" (say $n$ dimensional), the standard is often that all vectors are "columns", ...
- 218
0
votes
Why use transpose of nabla in gradient descent
Note that if f is scalar, you need an inner product to make a scalar on the right hand side. Generally nabla will be a vector
- 5,176
0
votes
Simulated Annealing Parameter Tuning
Just looking through the code: SA function is always called with the same value of temperature, $T=8$. The statement ...
- 1,580
2
votes
Accepted
Combinations from different sets with weightings
This can be expressed as a binary linear program. Provided the total number of unique combinations is not prohibitively large, it can be solved efficiently. When multiple solutions exist, usually ...
- 287k
2
votes
Why do we use Acquisition Functions?
In uncertain scenarios, where Bayesian optimization is employed, we are facing the exploration-exploitation tradeoff. The acquisition function is a solution for it. As you noticed, Bayesian ...
Tim♦
- 113k
0
votes
Examples in the Real World where Evolutionary Algorithms/Genetic Algorithms Outperform other Classes of Optimization Algorithms
It's still an open field of research, but there are many examples. Hyperparameter tuning is an example of a noisy and expensive function that can be handled by evolutionary algorithms. Neural ...
Tim♦
- 113k
0
votes
What does it mean when all gradients of a neural network are 0?
Gradients all equal to zero does not necessarily imply any problem with the network. Both minima and maxima occur where the gradient is zero. So it’s possible that your network has arrived at a local ...
- 78.2k
-2
votes
What does it mean when all gradients of a neural network are 0?
If you're using "tensorflow" and giving your model a batch make sure you put model(batch, training=True). If training isn't set to true then the gradient ...
Top 50 recent answers are included