# Questions tagged [optimization]

Use this tag for any use of optimization within statistics.

2,757 questions
Filter by
Sorted by
Tagged with
18 views

### Data analysis - ranking observation based on multiple parameters [closed]

I am currently looking for a method to rank my observation based on three numerical parameters. I would like to maximize two parameters and to minimize one in order to rank my observation. I have been ...
17 views

### Closed form expression for the 2-Wasserstein distance between generalized Gaussian distributions

Essentially the title - is there a closed form expression for the 2-Wasserstein distance (aka Frechet distance, Earth Mover's distance) between two generalized Gaussian distributions? The regular ...
23 views

### Implement Nesterov's acceleration for SVM

I am trying to implement Nestrov's acceleration gradient descent for SVM. The objective function I need to minimize is $$\frac{1}{2}\lVert Au-Bv\rVert_2^2$$ with constraints $\sum_{i}u_i=\sum_{j}v_j=1$...
69 views

### In linear models, why are we focused on BLUE rather than UMVUE?

It seems more natural to talk about UMVUE (following the basic statistical estimation theory). But when we turn to lm, we only care about BLUE, why? Are there any insurmountable difficulties here?
63 views

### MLE of multivariate normal distribution when the VCV matrix is full of equations

Short Version: Given a variance covariance matrix for my multivariate normal distribution where the entries are equations of other parameters, how do I find which of those parameter values maximizes ...
1 vote
13 views

### Determining Optimal Data Period / Time Span for Model Training

I'm seeking advice on determining the ideal time span for optimizing a weather forecast strategy using historical data without overfitting/underfitting our model. In pursuit of optimal performance and ...
1 vote
26 views

### Interrupt and restart optimization in R [closed]

I would like to use the Genetic Algorithm (GA) R-package for optimizing a land surface model. I need to be able to interrupt and restart the optimization at any given time because of node failures. In ...
40 views

### How to include parameter constraints in the Levenberg-Marquardt algorithm?

Every resource I found online doesn’t say what to do if the constraints aren’t satisfied. If my updated parameters is given by: $$\theta_{k+1} = \theta_k - (JJ^T + \lambda I)^{-1}Jr,$$ where J is the ...
34 views

### Model is nearly unidentifiable: large eigenvalue ratio in mixed logistic model

I am investigating effect of various "treatments" on various disease outcomes with univariate mixed logistic regression models corrected for covariates: ...
1 vote
19 views

1k views

### Why go through the trouble of expectation maximization and not use gradient descent?

In expectation maximization first a lower bound of the likelihood is found and then a 2 step iterative algorithm kicks in where first we try to find the weights (the probability that a data point ...
1 vote
54 views

### Perturbation for more stable convex optimization

I am thinking of adding some perturbation to my convex optimization problem. The idea is straight forward like below chart. Supposed you are solving $\text{argmax} f(x)$, we want to find an $x$ that'...
1 vote
23 views

### Do common implementations of mini-batch gradient descent violate the i.i.d assumption needed for unbiased estimation?

When we perform mini-batch GD, we estimate the true gradient: $$\nabla L = \frac{1}{N} \sum_i \nabla L_i$$ with: $$\nabla_B L = \frac{1}{B} \sum_{i \in B} \nabla L_i$$ where $B$ is the batch size. ...
26 views

### Optimising a multivariate table of counts based on marginals

I've been stuck on a problem for a very long time now so I decided to post on this forum for the first time. Although I am using code to perform this task, I believe it uses some statistics and I ...
39 views

### Overparameterization at the final layer of a neural network

I'm considering a multi-class classifying neural network with softmax on the final layer. Should one of the classes of the weight matrix of the final layer not be optimised over to account for the ...
57 views

### Is my custom loss function differentiable?

Consider the following loss function. loss = ( ( torch.where(d > threshold, torch.sqrt(d), 0) * t ) + ( torch.where(d <= threshold, (1 - d), 0) * (1 - t) ) ) ...
31 views

### Maxima in the dual of hard margin and soft margin SVMs?

The dual problem for hard margin SVM is: \begin{align*} &\max_{\alpha} \left( \sum_{i=1}^{N} \alpha_i - \frac{1}{2} \sum_{i=1}^{N} \sum_{j=1}^{N} \alpha_i \alpha_j y^{(i)} y^{(j)} \langle x^{(i)}, ...
9 views

### how to solve for wasserstein duality easily in a special case when 2-Wasserstein inequality constraint is binding

I was going through this nice paper ” A Simple and General Duality Proof for Wasserstein Distributionally Robust Optimization”, and one quick qu on applying Theorem 1 to my poject: What if in my ...
1 vote
31 views

### GLMM optimised with CG gives empty warning, diagnose() finds no problem

I am running the following code for a ZINB GLMM, using the glmmTMB package in R: ...
1 vote
129 views

### Newton's method and the Hessian for Softmax Logistic Regression

I'm having some trouble with optimising softmax regression via Newton's method. I'm not sure if the problem is arising with my equations for the Hessian and Gradient or with the code I've written. For ...
41 views

### Why does nlme::fdHess return NaNs when evaluated at 0? [closed]

Why does the finite-differences gradient/hessian approximation nlme::fdHess return NaNs when seeking the gradient/hessian at 0? <...
43 views

### Optimization metrics for a single-item recommendation system

I'm working on a recommendation system that's a bit different from ones I've built before. In particular, this system shows only the top item to the user, and the user can either click on it, dismiss ...
114 views

78 views

### Example where the initial random state of a logistic regression matters?

I am looking into how random initialisation of a model would impact final results after tuning. This is a well known problem for deep learning (NN or gbdt), notably with random initialisation and ...
13 views

### Co-occurrence matrix factorization ALS

Let's say I have an item-item co-occurrence matrix that I want to factorize. I'll thus only have item factors. Is it possible to learn the factors using ALS ? I can't see how given that we'd have to ...
17 views

### Divide weight decay values by the learning rate values in a grid search?

I have come across a paper where the authors do a grid search over hyperparameters. In particular, they tested different learning rates and weight decays. One thing that caught my attention was that ...
6 views

### Finding combination of bacteria that represents maximal functions encoded by them

I have the following data where rows represent functions and columns represent bacteria. I have 18 bacteria with a total of 160 functions (all funcitons are present in at least one bacteria). I want ...
1 vote
28 views

### Why I get spikes during training with vanilla gradient descent? [closed]

I developed my own NN toolbox, and it seems it works fine. But I am not sure why I get these spikes in my loss during training: I a training for a classification task of 2 inputs and 2 classes, ...
58 views

### Is optimization without function evaluations (and with only gradient evaluations) possible?

I am implementing an unconstrained optimization algorithm using gradient descent. I am evaluating a cost function at a given point, evaluating the gradient at this point, and selecting the next ...
13 views

### Measure Total Information About a System

Say I have a correlation matrix that measures how well certain components in a system track each other along some meaningful dimension, like condition state: So, for example, component 1 and ...
12 views

### How to speed up aftreg estimation?

I am performing an Accelerated Failure Time (AFT) regression data on a panel setting (multiple records per observation with time-dependent variables) using the eha ...
30 views

### Learning over non-independent joint distributions

For integer $n\geq 1$, I have a "goodness" function $f_n(F)$ that takes as input a given joint CDF $F$ of $n$ variables, and spits out a number in $[0,1]$ on how "good" $F$ is. The ...