Questions tagged [optimization]
Use this tag for any use of optimization within statistics.
1,866
questions
0
votes
0answers
7 views
Minimize variance of sum of weights in clusters
Let us assume we have $n$ products, with $w$ weight for each. Group $n$ products in $k$ bags such that the weights of the bags have least variance.
Let there be $h$ bags. Let $h^{th}$ bag contain ...
0
votes
0answers
10 views
glmer model - allFit function
I am conflicted over the results of the allFit() for my glmer model, but this has happened with lmer as well. What do you normally do when all the optimizers, except one, give you a very similar ...
2
votes
0answers
31 views
If $\ell_0$ regularization can be done via the proximal operator, why are people still using LASSO?
I have just learned that a general framework in constrained optimization is called "proximal gradient optimization". It is interesting that the $\ell_0$ "norm" is also associated with a proximal ...
4
votes
1answer
47 views
Density estimation as an optimization problem
Density estimation is the estimation of a probability density function from observed data. Can some of the common approaches to density estimation, such as kernel density estimation, be formulated as ...
0
votes
0answers
13 views
Bayesian update vs optimization in multivariate case
Say I have a multivariate normal vector
$r$~$N(\mu , \Sigma )$
and I observe that
$ y \equiv Pr + \epsilon = Q$
where $P$ is a matrix and $Q$ a vector and $\epsilon$~$N(0 , \Omega )$.
Now I ...
1
vote
0answers
6 views
Particle Swarm Optimisation Explained
Is anyone able to provide an intuitive explanation of how particle swarm optimisation works? For example how to minimise a function f(x,y,z). Also, does particle swarm optimisation work for multi-...
1
vote
1answer
11 views
Quadratic programming and interpretation of dual solution (Lagrangian)
Note: this question is about a common data science problem, but I am solving it using a specific piece of software. I believe the problem is common enough that these principles will be common across ...
0
votes
1answer
25 views
Similar loss, different results
I have trained multiple CNNs for image classification.
I suspect there is something wrong with my training pipeline, since many of my experiments get very similar training loss at the end of training, ...
1
vote
2answers
56 views
Is there an algorithm for finding the 10 best combinations from a list of 50 parts? [closed]
I've been thinking about a problem that seems pretty generic but I can't seem to find a solution for..
I have a list of 50 values. I need to make 10 groups which are as uniform as possible with ...
0
votes
0answers
8 views
How to enforce smoothness in guided image filtering techniques ? Any preferable model?
Which one (or more) of these three minimization models is the appropriate way to enforce smoothness in guided filtering framework ?
\begin{eqnarray}
%\begin{aligned}
& \sum\limits_{q \in {N}(p)} {\...
0
votes
0answers
20 views
Measure-agnostic learning?
I am studying metrics for evaluation of various learners in a multilabel classification setting. There seem to be more than 10 various measures, leaving me in some kind of doubt which metric to select,...
0
votes
0answers
18 views
Policy optimisation of a neural network which itself depends on the policy
I have a problem I've been thinking about recently and I am just completely stuck with regards to thinking of an appropriate way to tackle it.
The set-up is similar to the typical reinforcement ...
0
votes
1answer
20 views
Setting bound constraints in L-BFGS?
How does one choose the bounding constraints for the parameters in L-BFGS? Should these be viewed as a hyperparameter to be chosen subject to a criteria or do they arise as constraints in the typical "...
1
vote
0answers
16 views
A maximization problem involving random variables: a special case
Consider random variables $X$ and $Y$ that are jointly normally distributed,
$$
\begin{pmatrix} X \\ Y \end{pmatrix} \sim \mathcal{N}
\left[
\begin{pmatrix} \color{blue}0 \\ \mu_Y \end{pmatrix} ,
\...
2
votes
0answers
17 views
Minimization of the asymptotic variance in MCMC
Suppose $(X_n)_{n\in\mathbb N_0}$ is a Markov chain generated by the Metropolis-Hastings algorithm. Assume $(X_n)_{n\in\mathbb N_0}$ is stationary and consider the ergodic averages $$A_n:=\frac1n\sum_{...
0
votes
0answers
26 views
Optimal value for multiple input
I run an experiment in which every second I record values of area, circularity, and elongation (there will be probably more variables in the future).
I want to find in which second there are the ...
0
votes
0answers
13 views
Gradient on subset of training data is proportional to the true gradient?
I have been thinking of proving the following:
Prove that the gradient calculated on a random subset of a training
set on average is proportional to the true gradient.
However, proving is not my ...
0
votes
0answers
28 views
Fitting flexible spline using ODEs
I'm fitting a series of ordinary differential equations (describing movement through disease states: susceptible, infected, recovered) to weekly counts of a disease through time. I'm solving the ODEs ...
0
votes
0answers
21 views
How to create a variance-covariance matrix for forecasted fantasy basketball scores?
I have three basketball players who have played in games together and I want to find a Variance-Covariance matrix that will be as accurate as possible for their fantasy points in an upcoming game. My ...
0
votes
1answer
24 views
Genetic Algorithms for Feature Selection
Is anyone able to provide a simple explanation of how genetic algorithms can be used for feature selection in machine learning?
0
votes
1answer
31 views
How can we conclude that an optimization algorithm is better than another one for a problem at hand
When we test a new optimization algorithm for a particular problem at hand, what the process that we need to do?For example, do we need to run the algorithm several times, and pick a best performance,...
0
votes
1answer
45 views
Different optimization behaviours on delta vs non-delta targets
I built a simple classification model that is required to predict a probability distribution for a set of two available classes.
The target distributions are not necessarily delta distributions. i.e ...
1
vote
0answers
19 views
A maximization problem involving random variables
Consider random variables $X$ and $Y$ that are jointly normally distributed,
$$
\begin{pmatrix} X \\ Y \end{pmatrix} \sim \mathcal{N}
\left[
\begin{pmatrix} \mu_X \\ \mu_Y \end{pmatrix} ,
\begin{...
0
votes
0answers
19 views
Assessing the size of a cone by the singular values of $M$
Suppose I work with vectors from a high dimensional space with $100<N<1000$, e.g. word-embeddings. Say I have, already selected $R$ vectors, with $R\simeq10$, which form a matrix $M \in \mathbb{...
1
vote
0answers
24 views
How to deal with 'division by zero' and 'logarithm of zero' in simulations for finding the maximum log likelihood
I am trying to run a program which generates data from various covariate distributions, and finds the maximum likelihood estimator by explicitly maximising the log likelihood function. I am ...
0
votes
0answers
22 views
In semantic segmentation using Fully convolutional networks, why is Cross Entropy loss preferred over L1 or L2 losses?
I am training a fully convolutional network with Encoder-Decoder architecture for the task of Image Segmentation and currently am using the Binary Cross Entropy loss for foreground/background ...
2
votes
1answer
29 views
Support Vector Machine: identifying support vectors and kernel linear separability
I went through the MIT Artificial Intelligence lecture on Support Vector Machines by Professor Patrick Winston: https://www.youtube.com/watch?v=_PwhiWxHK8o
I've got a couple of questions. Would be ...
0
votes
0answers
43 views
Time complexity of batch gradient descent
I am read http://papers.nips.cc/paper/4937-accelerating-stochastic-gradient-descent-using-predictive-variance-reduction.pdf paper. It states that "Due to the poor condition number, the standard batch ...
0
votes
0answers
16 views
How AR parameters are transformed in `stats::arima` function?
There is a transform.pars argument in stats::arima function. This is a part of the explanation:
if true, the AR parameters ...
1
vote
1answer
20 views
What is the reasoning behind the default eps value for the 'SLSQP' method in SciPy's minimize function? [closed]
From here:
https://docs.scipy.org/doc/scipy/reference/optimize.minimize-slsqp.html#optimize-minimize-slsqp
It says that the default value for the eps option is <...
0
votes
1answer
22 views
class_weight = 'balanced' if GridSearch on unbalanced data set?
I'm trying to optimize the hyperparameters of an SVM. I have an unbalanced data set with more than two classes.
In some classes very many samples are included in others very few.
Using GridSearchCV, I ...
1
vote
0answers
39 views
Optimal proposal for the Metropolis-Hastings algorithm using Tierney's theorem
Let
$(E,\mathcal E,\lambda)$ be a measure space
$p:E\to[0,\infty)$ be $\mathcal E$-measurable with $$c:=\lambda p\in(0,\infty)$$ and $$\mu:=\underbrace{\frac1cp}_{=:\:\tilde p}\lambda$$
$q:E^2\to[0,\...
1
vote
1answer
32 views
K-Means clustering: optimal clusters for common data sets
I use scikit-learn to get IRIS and WINE clusters for evaluating an algorithm for K-means clustering. The K-means algorithm is a heuristic algorithm for solving the "minimum-sum-of-squares-clustering (...
1
vote
0answers
27 views
Can ADVI (Variational Inference) Induce Weak Multi Modality in a system with Uniform Priors, if a Gaussian Variational Family is Used
Question Set Up
If I have a weakly multi modal (see below in the edit) target posterior distribution which I am aiming to approximate using ADVI (Automatic Differentiation Variational Inference) with ...
1
vote
1answer
34 views
Why is the following choice of factor loadings optimal in two-state MLE for factor analysis?
Suppose we have $n$, $p$-dimensional, samples
$\overrightarrow{X_i} \sim \mathcal{N}(\mu, \Psi+\mathbf{w^Tw})$. $\Psi$ is a diagonal matrix of specific variances, while $\mathbf{w^Tw}$ composes the ...
0
votes
0answers
18 views
neural networks and optimization problems in general
Neural networks are efficient at solving optimization problems.
The topic of optimization problems is divided into linear and nonlinear problems and in linear and nonlinear conditions. I just wonder ...
0
votes
0answers
20 views
Solving constrained optimization problems with Adam
The adam algorithm has been very successful for solving non-convex optimization problems that appear in deep learning. Are there ways to extend adam to solve constrained optimization problems? Among ...
0
votes
1answer
15 views
How to compare distributions of values by the way they cluster along a line?
This is an optimization problem in Sudoku. I use a very fast brute force recursive fill-and-backtrack algorithm to count the number of solutions. This proceeds from the top-left to the bottom-right ...
0
votes
0answers
83 views
Variance reduction of an estimator arising from the marginal destribution of a Metropolis-Hastings chain
Let
$(E,\mathcal E,\lambda)$ and $(E',\mathcal E',\lambda')$ be measure spaces
$f\in L^2(\lambda)$
$I$ be a finite nonempty set
$\varphi_i:E'\to E$ be bijective $(\mathcal E',\mathcal E)$-measurable ...
3
votes
1answer
41 views
MLE when variance of residuals is null (y is a linear combination of x)
Suppose I have the following model to be estimated via MLE assuming normal errors $y_{t}=x_{t} \beta +e_{t}$ with $e=N(0,\sigma^{2})$, where $y, x$ are matrixes and $\beta$ is a vector, so $\sigma^{2}$...
0
votes
0answers
10 views
NNLS convergence plot
I am trying to understand how I can see the convergence of nnls optimization on a scatter plot. I have about 10 equations with 8 unknowns (a1, a2, a3 … a8) in the following form, a1 + 25*a2 = 100, a5 =...
1
vote
1answer
31 views
In gradient descent, could higher order gradients help to escape non global minima?
Im new to optimization so sorry if this question is ridiculous.
In gradient descent/ascent based optimization, one big problem seems to be getting stuck in a local minimum randomly. Ways in which I ...
0
votes
1answer
43 views
how deal with non-negatively of error in the model?
I am trying to solve a quadratic program in R. Since my matrix Dmat is not positive definite I am using dykstra to solve it. my model is:
...
0
votes
0answers
24 views
Update rule for gradient descent with momentum
I am really confused about applying gradient descent with momentum. The trusted resources which I use for learning about AI have different information.
CS231n says to use momentum like this,
Same ...
1
vote
0answers
82 views
Loss functions for Regression task
I am trying to understand the idea of Loss functions For Regression Task perfectly.
I have read many textbooks and articles, and I came up with questions related to this subject.
Several different ...
2
votes
1answer
26 views
The correct implementation of momentum method and NAG
Recently started a Coursera course on Deep Learning. In the optimization video, momentum and NAG were not very clearly explained so, I searched and came across the paper On the importance of ...
0
votes
1answer
20 views
How to modify RMSE loss function to adopt for integer valued predictions, using a Neural Network?
Context: Prediction of dependent variables like Age, Siblings, Children, etc (which are not categorical, but bounded, and integer-valued) from a dataset using Neural Network. I'm experimenting with a ...
3
votes
0answers
47 views
Beneficial dimension for 2nd order modelling in SGD optimization?
There are currently mostly used first order methods in SGD optimizers, second order are often seen too costly as e.g. full Hessian has size $D^2$ in dimension $D$.
But we don't need full Hessian - ...
0
votes
1answer
27 views
Transformation of Uniform Distribution to Real Number Line in ADVI
In the Automatic Differentiation Variational Inference (ADVI) paper, the authors claim to solve the VI problem in a transformed parameter space, which is over $\mathbb{R}$, in order to simplify the ...
0
votes
1answer
43 views
How to find better solutions for the k-means problem than by using the k-means/k-means++ algorithm?
The $k$-means problem in its common form can be stated as follows:
Given a data set $\mathcal{X}=x_1, ..., x_n$ consisting of $d$-dimensional vectors find a set $C = c_1,...,c_k$ of $d$-dimensional ...
