Questions tagged [optimization]

Use this tag for any use of optimization within statistics.

Filter by
Sorted by
Tagged with
0 votes
0 answers
18 views

Data analysis - ranking observation based on multiple parameters [closed]

I am currently looking for a method to rank my observation based on three numerical parameters. I would like to maximize two parameters and to minimize one in order to rank my observation. I have been ...
Lews's user avatar
  • 1
0 votes
0 answers
17 views

Closed form expression for the 2-Wasserstein distance between generalized Gaussian distributions

Essentially the title - is there a closed form expression for the 2-Wasserstein distance (aka Frechet distance, Earth Mover's distance) between two generalized Gaussian distributions? The regular ...
Shashank Gupta's user avatar
2 votes
0 answers
23 views

Implement Nesterov's acceleration for SVM

I am trying to implement Nestrov's acceleration gradient descent for SVM. The objective function I need to minimize is $$\frac{1}{2}\lVert Au-Bv\rVert_2^2$$ with constraints $\sum_{i}u_i=\sum_{j}v_j=1$...
struggleinmath's user avatar
2 votes
1 answer
69 views

In linear models, why are we focused on BLUE rather than UMVUE?

It seems more natural to talk about UMVUE (following the basic statistical estimation theory). But when we turn to lm, we only care about BLUE, why? Are there any insurmountable difficulties here?
ChS's user avatar
  • 83
0 votes
0 answers
63 views

MLE of multivariate normal distribution when the VCV matrix is full of equations

Short Version: Given a variance covariance matrix for my multivariate normal distribution where the entries are equations of other parameters, how do I find which of those parameter values maximizes ...
A Friendly Fish's user avatar
1 vote
0 answers
13 views

Determining Optimal Data Period / Time Span for Model Training

I'm seeking advice on determining the ideal time span for optimizing a weather forecast strategy using historical data without overfitting/underfitting our model. In pursuit of optimal performance and ...
RezAm's user avatar
  • 111
1 vote
1 answer
26 views

Interrupt and restart optimization in R [closed]

I would like to use the Genetic Algorithm (GA) R-package for optimizing a land surface model. I need to be able to interrupt and restart the optimization at any given time because of node failures. In ...
Christian's user avatar
2 votes
0 answers
40 views

How to include parameter constraints in the Levenberg-Marquardt algorithm?

Every resource I found online doesn’t say what to do if the constraints aren’t satisfied. If my updated parameters is given by: $$\theta_{k+1} = \theta_k - (JJ^T + \lambda I)^{-1}Jr, $$ where J is the ...
THAT'S MY QUANT MY QUANTITATIV's user avatar
0 votes
0 answers
34 views

Model is nearly unidentifiable: large eigenvalue ratio in mixed logistic model

I am investigating effect of various "treatments" on various disease outcomes with univariate mixed logistic regression models corrected for covariates: ...
JED HK's user avatar
  • 347
1 vote
1 answer
19 views

How to diagonalise when there is less parameters to estimate than data in the Levenberg-Marquardt algorithm

I am trying to calibrate a Heston Model with 100 call options using this paper https://arxiv.org/pdf/1511.08718.pdf. In algorithm 4.1 on page 18, they define the dampening factor as: $$\mu_0 = \omega \...
THAT'S MY QUANT MY QUANTITATIV's user avatar
0 votes
1 answer
41 views

Optimization and algorithms

I'm going over my practice midterm, and the last question has me stuck. It goes: Let $n \geq 1$ be an integer and let A be a symmetric $n \times n$ matrix (not necessary positive definite) for which ...
Albibi's user avatar
  • 1
0 votes
0 answers
36 views

What is the optimal number of trials in Optuna with TPESampler?

Just for context, I am using optuna to optimize hyperparameters of BERT model. I limited all my hyperparameter values to discrete and there are around 2000 possible combinations. I was wondering if ...
BloodthirstyPlatypus's user avatar
0 votes
0 answers
41 views

What is the size of Procrustes matrix?

Is there a contradiction in the presentations of orthogonal Procrustes rotation problem? In Matrix Computations by Golub&Van Loan (1) the orthogonal Procrustes problem is posed as an optimal ...
hamsa's user avatar
  • 21
0 votes
0 answers
15 views

For a stochastic system, should we optimize its time average cost or total cost? What is the difference between the two goals?

For a stochastic system, the time average cost like: $ \lim_{T \to \infty}\frac{1}{T}\sum_{t=0}^{T-1}\mathbb{E}\{cost(t)\}$ and total cost like: $ \lim_{T \to \infty}\sum_{t=0}^{T-1}\mathbb{E}\{cost(t)...
Cathy's user avatar
  • 43
0 votes
0 answers
17 views

what would be a good optimizer for black-box optimization in a unstable environment

My experiment is based on an online ab test system. A typical situation is that, I generate some candidates by some optimizer (CEM, GP etc) and push them to different experiment buckets (around 10). ...
G_cy's user avatar
  • 145
0 votes
0 answers
17 views

log-log regression as reward function in optimization problem

Consider the model $\hat{y}_t = e^{\text{trend} + \text{seasonality}} \prod_k^K x_{k, t}^{b_k}$ where $K$ denotes different investment alternatives. You can think that trend and seasonality are ...
pete lewis's user avatar
2 votes
1 answer
17 views

How to Normalise a loss function containing actual values?

I defined a loss function that I am trying to optimise consisting of two elements. let's say a and b: w1 * (a_actual - a_measured)^2 + w2* (b_actual - b_measured)^2 I am trying to simplify the ...
user avatar
0 votes
0 answers
12 views

What statistical test should be used to assess for differences in knowledge retention of the same group of participants at 4 different time points?

I am conducting a study to assess knowledge retention rates among a group of students over a period of time. Participants took a quiz at baseline, after a lesson, 3 months later, and then 1 year later....
Shane's user avatar
  • 1
0 votes
0 answers
9 views

Impact analysis of individual independent components

I need help in understanding how we can analyse the impact of an individual independent variable (campaign media type in this case) on a dependent variable (sales performance here) when not using the ...
Mr. Confused's user avatar
1 vote
0 answers
21 views

Finding a condition that maximizes the conditional correlation between two random variables

Let’s say I have $n$ independent and identically distributed samples that is, $(X_1,Y_1)$, $(X_2,Y_2)$, ... $(X_n,Y_n)$ which is my dataset. I want to find a condition on $X$ such that the ...
Akshat Joshi's user avatar
1 vote
1 answer
38 views

Reducing the number of observables in Factor Analysis

I have $k$ observables (e.g. questions in a questionnaire) and $n$ observational units (e.g. respondents of the questionnaire). Let's call the observation matrix $X$. Let's assume I have performed a ...
Aleksejs Fomins's user avatar
0 votes
0 answers
8 views

Deep NN with positive partial derivative

Let's assume we are given a FFNN of type $$F: \mathbb{R}^n \times \mathbb{R} \rightarrow \mathbb{R}, \quad (x_1,...,x_{n+1}) \mapsto y$$ We assume the generic architecture (of depth $H$) $$a^{l+1}=\...
NicAG's user avatar
  • 171
11 votes
2 answers
1k views

Why go through the trouble of expectation maximization and not use gradient descent?

In expectation maximization first a lower bound of the likelihood is found and then a 2 step iterative algorithm kicks in where first we try to find the weights (the probability that a data point ...
figs_and_nuts's user avatar
1 vote
1 answer
54 views

Perturbation for more stable convex optimization

I am thinking of adding some perturbation to my convex optimization problem. The idea is straight forward like below chart. Supposed you are solving $\text{argmax} f(x) $, we want to find an $x$ that'...
Matt Frank's user avatar
1 vote
0 answers
23 views

Do common implementations of mini-batch gradient descent violate the i.i.d assumption needed for unbiased estimation?

When we perform mini-batch GD, we estimate the true gradient: $$\nabla L = \frac{1}{N} \sum_i \nabla L_i$$ with: $$\nabla_B L = \frac{1}{B} \sum_{i \in B} \nabla L_i$$ where $B$ is the batch size. ...
ado sar's user avatar
  • 452
0 votes
0 answers
26 views

Optimising a multivariate table of counts based on marginals

I've been stuck on a problem for a very long time now so I decided to post on this forum for the first time. Although I am using code to perform this task, I believe it uses some statistics and I ...
taek's user avatar
  • 3
0 votes
1 answer
39 views

Overparameterization at the final layer of a neural network

I'm considering a multi-class classifying neural network with softmax on the final layer. Should one of the classes of the weight matrix of the final layer not be optimised over to account for the ...
oweydd's user avatar
  • 165
0 votes
0 answers
57 views

Is my custom loss function differentiable?

Consider the following loss function. loss = ( ( torch.where(d > threshold, torch.sqrt(d), 0) * t ) + ( torch.where(d <= threshold, (1 - d), 0) * (1 - t) ) ) ...
Adel's user avatar
  • 285
0 votes
1 answer
31 views

Maxima in the dual of hard margin and soft margin SVMs?

The dual problem for hard margin SVM is: \begin{align*} &\max_{\alpha} \left( \sum_{i=1}^{N} \alpha_i - \frac{1}{2} \sum_{i=1}^{N} \sum_{j=1}^{N} \alpha_i \alpha_j y^{(i)} y^{(j)} \langle x^{(i)}, ...
something something's user avatar
0 votes
0 answers
9 views

how to solve for wasserstein duality easily in a special case when 2-Wasserstein inequality constraint is binding

I was going through this nice paper ” A Simple and General Duality Proof for Wasserstein Distributionally Robust Optimization”, and one quick qu on applying Theorem 1 to my poject: What if in my ...
numpynp's user avatar
  • 21
1 vote
0 answers
31 views

GLMM optimised with CG gives empty warning, diagnose() finds no problem

I am running the following code for a ZINB GLMM, using the glmmTMB package in R: ...
Barbara Perez de Araújo's user avatar
1 vote
1 answer
129 views

Newton's method and the Hessian for Softmax Logistic Regression

I'm having some trouble with optimising softmax regression via Newton's method. I'm not sure if the problem is arising with my equations for the Hessian and Gradient or with the code I've written. For ...
oweydd's user avatar
  • 165
3 votes
1 answer
41 views

Why does nlme::fdHess return NaNs when evaluated at 0? [closed]

Why does the finite-differences gradient/hessian approximation nlme::fdHess return NaNs when seeking the gradient/hessian at 0? <...
kara890's user avatar
  • 129
0 votes
0 answers
43 views

Optimization metrics for a single-item recommendation system

I'm working on a recommendation system that's a bit different from ones I've built before. In particular, this system shows only the top item to the user, and the user can either click on it, dismiss ...
Kyranstar's user avatar
  • 113
3 votes
0 answers
114 views

Euclidean and geodesic distance have different gradients. Does mixing the two concepts impair triplet learning?

The triplet loss is defined by Florian Schroff, Dmitry Kalenichenko, James Philbin in "FaceNet: A Unified Embedding for Face Recognition and Clustering" as $$ \mathcal L = \sum_\mathcal T \...
Sycorax's user avatar
  • 89k
0 votes
0 answers
28 views

Optimization problem involving VAR

I'm looking to solve, from the paper https://arxiv.org/pdf/2210.02176.pdf, $$\min_{g\in VAR(p)} \sum_{z\in \mathcal{Z}} (f( h_{x}(z))-g(z))^2\pi_{x}(z)$$ where $\mathcal{Z}\in \\{0,1\\}^{N\times W}$ ...
Kashif's user avatar
  • 507
0 votes
0 answers
6 views

Using validation data in optimization scenarios e.g. with genetic algorithms

I'm not too familiar with optimization algorithms e.g. genetic algorithms and I'm wondering whether it makes sense to employ a validation set in this context similarly to when we train a supervised ...
James Arten's user avatar
2 votes
1 answer
36 views

Censored Regression model

I am considering the following censored regression model. I have a question about my R code. Then I obtain likelihood function as follow. Then i obtain log-likelihood function. ...
Hiroki's user avatar
  • 31
2 votes
2 answers
67 views

How do I obtain the primal and dual for the estimator $\min _\beta\left[\|\beta\|^2+\sum_{i=1}^n \xi_i^2\right]$ s.t. $\xi_i=y_i-h(x_i)^\top \beta$?

I am working on a statistical learning exercise that requires some knowledge of convex optimization which I am unfortunately lacking. Consider the linear regression model $$y_i=h(x_i)^\top\beta+\...
Leon's user avatar
  • 41
-1 votes
2 answers
78 views

Example where the initial random state of a logistic regression matters?

I am looking into how random initialisation of a model would impact final results after tuning. This is a well known problem for deep learning (NN or gbdt), notably with random initialisation and ...
Lucas Morin's user avatar
  • 1,544
0 votes
0 answers
13 views

Co-occurrence matrix factorization ALS

Let's say I have an item-item co-occurrence matrix that I want to factorize. I'll thus only have item factors. Is it possible to learn the factors using ALS ? I can't see how given that we'd have to ...
Koowah's user avatar
  • 1
0 votes
0 answers
17 views

Divide weight decay values by the learning rate values in a grid search?

I have come across a paper where the authors do a grid search over hyperparameters. In particular, they tested different learning rates and weight decays. One thing that caught my attention was that ...
Mikhail Petrov's user avatar
0 votes
0 answers
6 views

Finding combination of bacteria that represents maximal functions encoded by them

I have the following data where rows represent functions and columns represent bacteria. I have 18 bacteria with a total of 160 functions (all funcitons are present in at least one bacteria). I want ...
Arijit's user avatar
  • 71
1 vote
0 answers
28 views

Why I get spikes during training with vanilla gradient descent? [closed]

I developed my own NN toolbox, and it seems it works fine. But I am not sure why I get these spikes in my loss during training: I a training for a classification task of 2 inputs and 2 classes, ...
z_tjona's user avatar
  • 119
4 votes
2 answers
58 views

Is optimization without function evaluations (and with only gradient evaluations) possible?

I am implementing an unconstrained optimization algorithm using gradient descent. I am evaluating a cost function at a given point, evaluating the gradient at this point, and selecting the next ...
Joaquin Rapela's user avatar
0 votes
0 answers
13 views

Measure Total Information About a System

Say I have a correlation matrix that measures how well certain components in a system track each other along some meaningful dimension, like condition state: So, for example, component 1 and ...
coolhand's user avatar
  • 199
0 votes
1 answer
12 views

How to speed up aftreg estimation?

I am performing an Accelerated Failure Time (AFT) regression data on a panel setting (multiple records per observation with time-dependent variables) using the eha ...
Babak Fi Foo's user avatar
0 votes
0 answers
30 views

Learning over non-independent joint distributions

For integer $n\geq 1$, I have a "goodness" function $f_n(F)$ that takes as input a given joint CDF $F$ of $n$ variables, and spits out a number in $[0,1]$ on how "good" $F$ is. The ...
AspiringMat's user avatar
0 votes
1 answer
38 views

Cross validation and hyperparameters tuning [duplicate]

I have few questions concerning the selection of hyperparameters for predictions in cross validation. If I understand well, during the CV, you just create folds (inner and outer for a nested CV), and ...
Nicolas's user avatar
  • 13
1 vote
0 answers
18 views

Optimizing Algorithmic Trading Parameters: A Quest for Maximizing Expected Value

I am engaged in algorithmic trading, employing specialized models that utilize various parameters to signal trading opportunities. Once I get a signal, I could execute a trade and everything is ...
David's user avatar
  • 11

1
2 3 4 5
56