The Stack Overflow podcast is back! Listen to an interview with our new CEO.

Questions tagged [optimization]

Use this tag for any use of optimization within statistics.

Filter by
Sorted by
Tagged with
0
votes
0answers
7 views

Minimize variance of sum of weights in clusters

Let us assume we have $n$ products, with $w$ weight for each. Group $n$ products in $k$ bags such that the weights of the bags have least variance. Let there be $h$ bags. Let $h^{th}$ bag contain ...
0
votes
0answers
10 views

glmer model - allFit function

I am conflicted over the results of the allFit() for my glmer model, but this has happened with lmer as well. What do you normally do when all the optimizers, except one, give you a very similar ...
2
votes
0answers
31 views

If $\ell_0$ regularization can be done via the proximal operator, why are people still using LASSO?

I have just learned that a general framework in constrained optimization is called "proximal gradient optimization". It is interesting that the $\ell_0$ "norm" is also associated with a proximal ...
4
votes
1answer
47 views

Density estimation as an optimization problem

Density estimation is the estimation of a probability density function from observed data. Can some of the common approaches to density estimation, such as kernel density estimation, be formulated as ...
0
votes
0answers
13 views

Bayesian update vs optimization in multivariate case

Say I have a multivariate normal vector $r$~$N(\mu , \Sigma )$ and I observe that $ y \equiv Pr + \epsilon = Q$ where $P$ is a matrix and $Q$ a vector and $\epsilon$~$N(0 , \Omega )$. Now I ...
1
vote
0answers
6 views

Particle Swarm Optimisation Explained

Is anyone able to provide an intuitive explanation of how particle swarm optimisation works? For example how to minimise a function f(x,y,z). Also, does particle swarm optimisation work for multi-...
1
vote
1answer
11 views

Quadratic programming and interpretation of dual solution (Lagrangian)

Note: this question is about a common data science problem, but I am solving it using a specific piece of software. I believe the problem is common enough that these principles will be common across ...
0
votes
1answer
25 views

Similar loss, different results

I have trained multiple CNNs for image classification. I suspect there is something wrong with my training pipeline, since many of my experiments get very similar training loss at the end of training, ...
1
vote
2answers
56 views

Is there an algorithm for finding the 10 best combinations from a list of 50 parts? [closed]

I've been thinking about a problem that seems pretty generic but I can't seem to find a solution for.. I have a list of 50 values. I need to make 10 groups which are as uniform as possible with ...
0
votes
0answers
8 views

How to enforce smoothness in guided image filtering techniques ? Any preferable model?

Which one (or more) of these three minimization models is the appropriate way to enforce smoothness in guided filtering framework ? \begin{eqnarray} %\begin{aligned} & \sum\limits_{q \in {N}(p)} {\...
0
votes
0answers
20 views

Measure-agnostic learning?

I am studying metrics for evaluation of various learners in a multilabel classification setting. There seem to be more than 10 various measures, leaving me in some kind of doubt which metric to select,...
0
votes
0answers
18 views

Policy optimisation of a neural network which itself depends on the policy

I have a problem I've been thinking about recently and I am just completely stuck with regards to thinking of an appropriate way to tackle it. The set-up is similar to the typical reinforcement ...
0
votes
1answer
20 views

Setting bound constraints in L-BFGS?

How does one choose the bounding constraints for the parameters in L-BFGS? Should these be viewed as a hyperparameter to be chosen subject to a criteria or do they arise as constraints in the typical "...
1
vote
0answers
16 views

A maximization problem involving random variables: a special case

Consider random variables $X$ and $Y$ that are jointly normally distributed, $$ \begin{pmatrix} X \\ Y \end{pmatrix} \sim \mathcal{N} \left[ \begin{pmatrix} \color{blue}0 \\ \mu_Y \end{pmatrix} , \...
2
votes
0answers
17 views

Minimization of the asymptotic variance in MCMC

Suppose $(X_n)_{n\in\mathbb N_0}$ is a Markov chain generated by the Metropolis-Hastings algorithm. Assume $(X_n)_{n\in\mathbb N_0}$ is stationary and consider the ergodic averages $$A_n:=\frac1n\sum_{...
0
votes
0answers
26 views

Optimal value for multiple input

I run an experiment in which every second I record values of area, circularity, and elongation (there will be probably more variables in the future). I want to find in which second there are the ...
0
votes
0answers
13 views

Gradient on subset of training data is proportional to the true gradient?

I have been thinking of proving the following: Prove that the gradient calculated on a random subset of a training set on average is proportional to the true gradient. However, proving is not my ...
0
votes
0answers
28 views

Fitting flexible spline using ODEs

I'm fitting a series of ordinary differential equations (describing movement through disease states: susceptible, infected, recovered) to weekly counts of a disease through time. I'm solving the ODEs ...
0
votes
0answers
21 views

How to create a variance-covariance matrix for forecasted fantasy basketball scores?

I have three basketball players who have played in games together and I want to find a Variance-Covariance matrix that will be as accurate as possible for their fantasy points in an upcoming game. My ...
0
votes
1answer
24 views

Genetic Algorithms for Feature Selection

Is anyone able to provide a simple explanation of how genetic algorithms can be used for feature selection in machine learning?
0
votes
1answer
31 views

How can we conclude that an optimization algorithm is better than another one for a problem at hand

When we test a new optimization algorithm for a particular problem at hand, what the process that we need to do?For example, do we need to run the algorithm several times, and pick a best performance,...
0
votes
1answer
45 views

Different optimization behaviours on delta vs non-delta targets

I built a simple classification model that is required to predict a probability distribution for a set of two available classes. The target distributions are not necessarily delta distributions. i.e ...
1
vote
0answers
19 views

A maximization problem involving random variables

Consider random variables $X$ and $Y$ that are jointly normally distributed, $$ \begin{pmatrix} X \\ Y \end{pmatrix} \sim \mathcal{N} \left[ \begin{pmatrix} \mu_X \\ \mu_Y \end{pmatrix} , \begin{...
0
votes
0answers
19 views

Assessing the size of a cone by the singular values of $M$

Suppose I work with vectors from a high dimensional space with $100<N<1000$, e.g. word-embeddings. Say I have, already selected $R$ vectors, with $R\simeq10$, which form a matrix $M \in \mathbb{...
1
vote
0answers
24 views

How to deal with 'division by zero' and 'logarithm of zero' in simulations for finding the maximum log likelihood

I am trying to run a program which generates data from various covariate distributions, and finds the maximum likelihood estimator by explicitly maximising the log likelihood function. I am ...
0
votes
0answers
22 views

In semantic segmentation using Fully convolutional networks, why is Cross Entropy loss preferred over L1 or L2 losses?

I am training a fully convolutional network with Encoder-Decoder architecture for the task of Image Segmentation and currently am using the Binary Cross Entropy loss for foreground/background ...
2
votes
1answer
29 views

Support Vector Machine: identifying support vectors and kernel linear separability

I went through the MIT Artificial Intelligence lecture on Support Vector Machines by Professor Patrick Winston: https://www.youtube.com/watch?v=_PwhiWxHK8o I've got a couple of questions. Would be ...
0
votes
0answers
43 views

Time complexity of batch gradient descent

I am read http://papers.nips.cc/paper/4937-accelerating-stochastic-gradient-descent-using-predictive-variance-reduction.pdf paper. It states that "Due to the poor condition number, the standard batch ...
0
votes
0answers
16 views

How AR parameters are transformed in `stats::arima` function?

There is a transform.pars argument in stats::arima function. This is a part of the explanation: if true, the AR parameters ...
1
vote
1answer
20 views

What is the reasoning behind the default eps value for the 'SLSQP' method in SciPy's minimize function? [closed]

From here: https://docs.scipy.org/doc/scipy/reference/optimize.minimize-slsqp.html#optimize-minimize-slsqp It says that the default value for the eps option is <...
0
votes
1answer
22 views

class_weight = 'balanced' if GridSearch on unbalanced data set?

I'm trying to optimize the hyperparameters of an SVM. I have an unbalanced data set with more than two classes. In some classes very many samples are included in others very few. Using GridSearchCV, I ...
1
vote
0answers
39 views

Optimal proposal for the Metropolis-Hastings algorithm using Tierney's theorem

Let $(E,\mathcal E,\lambda)$ be a measure space $p:E\to[0,\infty)$ be $\mathcal E$-measurable with $$c:=\lambda p\in(0,\infty)$$ and $$\mu:=\underbrace{\frac1cp}_{=:\:\tilde p}\lambda$$ $q:E^2\to[0,\...
1
vote
1answer
32 views

K-Means clustering: optimal clusters for common data sets

I use scikit-learn to get IRIS and WINE clusters for evaluating an algorithm for K-means clustering. The K-means algorithm is a heuristic algorithm for solving the "minimum-sum-of-squares-clustering (...
1
vote
0answers
27 views

Can ADVI (Variational Inference) Induce Weak Multi Modality in a system with Uniform Priors, if a Gaussian Variational Family is Used

Question Set Up If I have a weakly multi modal (see below in the edit) target posterior distribution which I am aiming to approximate using ADVI (Automatic Differentiation Variational Inference) with ...
1
vote
1answer
34 views

Why is the following choice of factor loadings optimal in two-state MLE for factor analysis?

Suppose we have $n$, $p$-dimensional, samples $\overrightarrow{X_i} \sim \mathcal{N}(\mu, \Psi+\mathbf{w^Tw})$. $\Psi$ is a diagonal matrix of specific variances, while $\mathbf{w^Tw}$ composes the ...
0
votes
0answers
18 views

neural networks and optimization problems in general

Neural networks are efficient at solving optimization problems. The topic of optimization problems is divided into linear and nonlinear problems and in linear and nonlinear conditions. I just wonder ...
0
votes
0answers
20 views

Solving constrained optimization problems with Adam

The adam algorithm has been very successful for solving non-convex optimization problems that appear in deep learning. Are there ways to extend adam to solve constrained optimization problems? Among ...
0
votes
1answer
15 views

How to compare distributions of values by the way they cluster along a line?

This is an optimization problem in Sudoku. I use a very fast brute force recursive fill-and-backtrack algorithm to count the number of solutions. This proceeds from the top-left to the bottom-right ...
0
votes
0answers
83 views

Variance reduction of an estimator arising from the marginal destribution of a Metropolis-Hastings chain

Let $(E,\mathcal E,\lambda)$ and $(E',\mathcal E',\lambda')$ be measure spaces $f\in L^2(\lambda)$ $I$ be a finite nonempty set $\varphi_i:E'\to E$ be bijective $(\mathcal E',\mathcal E)$-measurable ...
3
votes
1answer
41 views

MLE when variance of residuals is null (y is a linear combination of x)

Suppose I have the following model to be estimated via MLE assuming normal errors $y_{t}=x_{t} \beta +e_{t}$ with $e=N(0,\sigma^{2})$, where $y, x$ are matrixes and $\beta$ is a vector, so $\sigma^{2}$...
0
votes
0answers
10 views

NNLS convergence plot

I am trying to understand how I can see the convergence of nnls optimization on a scatter plot. I have about 10 equations with 8 unknowns (a1, a2, a3 … a8) in the following form, a1 + 25*a2 = 100, a5 =...
1
vote
1answer
31 views

In gradient descent, could higher order gradients help to escape non global minima?

Im new to optimization so sorry if this question is ridiculous. In gradient descent/ascent based optimization, one big problem seems to be getting stuck in a local minimum randomly. Ways in which I ...
0
votes
1answer
43 views

how deal with non-negatively of error in the model?

I am trying to solve a quadratic program in R. Since my matrix Dmat is not positive definite I am using dykstra to solve it. my model is: ...
0
votes
0answers
24 views

Update rule for gradient descent with momentum

I am really confused about applying gradient descent with momentum. The trusted resources which I use for learning about AI have different information. CS231n says to use momentum like this, Same ...
1
vote
0answers
82 views

Loss functions for Regression task

I am trying to understand the idea of Loss functions For Regression Task perfectly. I have read many textbooks and articles, and I came up with questions related to this subject. Several different ...
2
votes
1answer
26 views

The correct implementation of momentum method and NAG

Recently started a Coursera course on Deep Learning. In the optimization video, momentum and NAG were not very clearly explained so, I searched and came across the paper On the importance of ...
0
votes
1answer
20 views

How to modify RMSE loss function to adopt for integer valued predictions, using a Neural Network?

Context: Prediction of dependent variables like Age, Siblings, Children, etc (which are not categorical, but bounded, and integer-valued) from a dataset using Neural Network. I'm experimenting with a ...
3
votes
0answers
47 views

Beneficial dimension for 2nd order modelling in SGD optimization?

There are currently mostly used first order methods in SGD optimizers, second order are often seen too costly as e.g. full Hessian has size $D^2$ in dimension $D$. But we don't need full Hessian - ...
0
votes
1answer
27 views

Transformation of Uniform Distribution to Real Number Line in ADVI

In the Automatic Differentiation Variational Inference (ADVI) paper, the authors claim to solve the VI problem in a transformed parameter space, which is over $\mathbb{R}$, in order to simplify the ...
0
votes
1answer
43 views

How to find better solutions for the k-means problem than by using the k-means/k-means++ algorithm?

The $k$-means problem in its common form can be stated as follows: Given a data set $\mathcal{X}=x_1, ..., x_n$ consisting of $d$-dimensional vectors find a set $C = c_1,...,c_k$ of $d$-dimensional ...