Questions tagged [optimization]
Use this tag for any use of optimization within statistics.
2,757
questions
0
votes
0
answers
18
views
Data analysis - ranking observation based on multiple parameters [closed]
I am currently looking for a method to rank my observation based on three numerical parameters. I would like to maximize two parameters and to minimize one in order to rank my observation. I have been ...
- 1
0
votes
0
answers
17
views
Closed form expression for the 2-Wasserstein distance between generalized Gaussian distributions
Essentially the title - is there a closed form expression for the 2-Wasserstein distance (aka Frechet distance, Earth Mover's distance) between two generalized Gaussian distributions? The regular ...
2
votes
0
answers
23
views
Implement Nesterov's acceleration for SVM
I am trying to implement Nestrov's acceleration gradient descent for SVM. The objective function I need to minimize is $$\frac{1}{2}\lVert Au-Bv\rVert_2^2$$ with constraints $\sum_{i}u_i=\sum_{j}v_j=1$...
2
votes
1
answer
69
views
In linear models, why are we focused on BLUE rather than UMVUE?
It seems more natural to talk about UMVUE (following the basic statistical estimation theory). But when we turn to lm, we only care about BLUE, why? Are there any insurmountable difficulties here?
- 83
0
votes
0
answers
63
views
MLE of multivariate normal distribution when the VCV matrix is full of equations
Short Version: Given a variance covariance matrix for my multivariate normal distribution where the entries are equations of other parameters, how do I find which of those parameter values maximizes ...
1
vote
0
answers
13
views
Determining Optimal Data Period / Time Span for Model Training
I'm seeking advice on determining the ideal time span for optimizing a weather forecast strategy using historical data without overfitting/underfitting our model. In pursuit of optimal performance and ...
- 111
1
vote
1
answer
26
views
Interrupt and restart optimization in R [closed]
I would like to use the Genetic Algorithm (GA) R-package for optimizing a land surface model. I need to be able to interrupt and restart the optimization at any given time because of node failures. In ...
- 13
2
votes
0
answers
40
views
How to include parameter constraints in the Levenberg-Marquardt algorithm?
Every resource I found online doesn’t say what to do if the constraints aren’t satisfied. If my updated parameters is given by:
$$\theta_{k+1} = \theta_k - (JJ^T + \lambda I)^{-1}Jr, $$ where J is the ...
0
votes
0
answers
34
views
Model is nearly unidentifiable: large eigenvalue ratio in mixed logistic model
I am investigating effect of various "treatments" on various disease outcomes with univariate mixed logistic regression models corrected for covariates:
...
- 347
1
vote
1
answer
19
views
How to diagonalise when there is less parameters to estimate than data in the Levenberg-Marquardt algorithm
I am trying to calibrate a Heston Model with 100 call options using this paper https://arxiv.org/pdf/1511.08718.pdf.
In algorithm 4.1 on page 18, they define the dampening factor as: $$\mu_0 = \omega \...
0
votes
1
answer
41
views
Optimization and algorithms
I'm going over my practice midterm, and the last question has me stuck. It goes:
Let $n \geq 1$ be an integer and let A be a symmetric $n \times n$ matrix (not necessary
positive definite) for which ...
- 1
0
votes
0
answers
36
views
What is the optimal number of trials in Optuna with TPESampler?
Just for context, I am using optuna to optimize hyperparameters of BERT model.
I limited all my hyperparameter values to discrete and there are around 2000 possible combinations.
I was wondering if ...
0
votes
0
answers
41
views
What is the size of Procrustes matrix?
Is there a contradiction in the presentations of orthogonal Procrustes rotation problem?
In Matrix Computations by Golub&Van Loan (1) the orthogonal Procrustes problem is posed as an optimal ...
- 21
0
votes
0
answers
15
views
For a stochastic system, should we optimize its time average cost or total cost? What is the difference between the two goals?
For a stochastic system, the time average cost like:
$ \lim_{T \to \infty}\frac{1}{T}\sum_{t=0}^{T-1}\mathbb{E}\{cost(t)\}$
and total cost like:
$ \lim_{T \to \infty}\sum_{t=0}^{T-1}\mathbb{E}\{cost(t)...
- 43
0
votes
0
answers
17
views
what would be a good optimizer for black-box optimization in a unstable environment
My experiment is based on an online ab test system. A typical situation is that, I generate some candidates by some optimizer (CEM, GP etc) and push them to different experiment buckets (around 10). ...
- 145
0
votes
0
answers
17
views
log-log regression as reward function in optimization problem
Consider the model $\hat{y}_t = e^{\text{trend} + \text{seasonality}} \prod_k^K x_{k, t}^{b_k}$
where $K$ denotes different investment alternatives. You can think that trend and seasonality are ...
2
votes
1
answer
17
views
How to Normalise a loss function containing actual values?
I defined a loss function that I am trying to optimise consisting of two elements. let's say a and b:
w1 * (a_actual - a_measured)^2 + w2* (b_actual - b_measured)^2
I am trying to simplify the ...
Peyman Shadmani
0
votes
0
answers
12
views
What statistical test should be used to assess for differences in knowledge retention of the same group of participants at 4 different time points?
I am conducting a study to assess knowledge retention rates among a group of students over a period of time. Participants took a quiz at baseline, after a lesson, 3 months later, and then 1 year later....
- 1
0
votes
0
answers
9
views
Impact analysis of individual independent components
I need help in understanding how we can analyse the impact of an individual independent variable (campaign media type in this case) on a dependent variable (sales performance here) when not using the ...
- 313
1
vote
0
answers
21
views
Finding a condition that maximizes the conditional correlation between two random variables
Let’s say I have $n$ independent and identically distributed samples that is, $(X_1,Y_1)$, $(X_2,Y_2)$, ... $(X_n,Y_n)$ which is my dataset. I want to find a condition on $X$ such that the ...
- 111
1
vote
1
answer
38
views
Reducing the number of observables in Factor Analysis
I have $k$ observables (e.g. questions in a questionnaire) and $n$ observational units (e.g. respondents of the questionnaire). Let's call the observation matrix $X$.
Let's assume I have performed a ...
- 1,791
0
votes
0
answers
8
views
Deep NN with positive partial derivative
Let's assume we are given a FFNN of type
$$F: \mathbb{R}^n \times \mathbb{R} \rightarrow \mathbb{R}, \quad (x_1,...,x_{n+1}) \mapsto y$$
We assume the generic architecture (of depth $H$)
$$a^{l+1}=\...
- 171
11
votes
2
answers
1k
views
Why go through the trouble of expectation maximization and not use gradient descent?
In expectation maximization first a lower bound of the likelihood is found and then a 2 step iterative algorithm kicks in where first we try to find the weights (the probability that a data point ...
- 2,385
1
vote
1
answer
54
views
Perturbation for more stable convex optimization
I am thinking of adding some perturbation to my convex optimization problem. The idea is straight forward like below chart. Supposed you are solving $\text{argmax} f(x) $, we want to find an $x$ that'...
- 133
1
vote
0
answers
23
views
Do common implementations of mini-batch gradient descent violate the i.i.d assumption needed for unbiased estimation?
When we perform mini-batch GD, we estimate the true gradient:
$$\nabla L = \frac{1}{N} \sum_i \nabla L_i$$
with:
$$\nabla_B L = \frac{1}{B} \sum_{i \in B} \nabla L_i$$
where $B$ is the batch size. ...
- 452
0
votes
0
answers
26
views
Optimising a multivariate table of counts based on marginals
I've been stuck on a problem for a very long time now so I decided to post on this forum for the first time. Although I am using code to perform this task, I believe it uses some statistics and I ...
- 3
0
votes
1
answer
39
views
Overparameterization at the final layer of a neural network
I'm considering a multi-class classifying neural network with softmax on the final layer. Should one of the classes of the weight matrix of the final layer not be optimised over to account for the ...
- 165
0
votes
0
answers
57
views
Is my custom loss function differentiable?
Consider the following loss function.
loss = ( ( torch.where(d > threshold, torch.sqrt(d), 0) * t ) + ( torch.where(d <= threshold, (1 - d), 0) * (1 - t) ) )
...
- 285
0
votes
1
answer
31
views
Maxima in the dual of hard margin and soft margin SVMs?
The dual problem for hard margin SVM is:
\begin{align*}
&\max_{\alpha} \left( \sum_{i=1}^{N} \alpha_i - \frac{1}{2} \sum_{i=1}^{N} \sum_{j=1}^{N} \alpha_i \alpha_j y^{(i)} y^{(j)} \langle x^{(i)}, ...
0
votes
0
answers
9
views
how to solve for wasserstein duality easily in a special case when 2-Wasserstein inequality constraint is binding
I was going through this nice paper ” A Simple and General Duality Proof for
Wasserstein Distributionally Robust Optimization”, and one quick qu on applying Theorem 1 to my poject:
What if in my ...
- 21
1
vote
0
answers
31
views
GLMM optimised with CG gives empty warning, diagnose() finds no problem
I am running the following code for a ZINB GLMM, using the glmmTMB package in R:
...
1
vote
1
answer
129
views
Newton's method and the Hessian for Softmax Logistic Regression
I'm having some trouble with optimising softmax regression via Newton's method. I'm not sure if the problem is arising with my equations for the Hessian and Gradient or with the code I've written.
For ...
- 165
3
votes
1
answer
41
views
Why does nlme::fdHess return NaNs when evaluated at 0? [closed]
Why does the finite-differences gradient/hessian approximation nlme::fdHess return NaNs when seeking the gradient/hessian at 0?
<...
- 129
0
votes
0
answers
43
views
Optimization metrics for a single-item recommendation system
I'm working on a recommendation system that's a bit different from ones I've built before. In particular, this system shows only the top item to the user, and the user can either click on it, dismiss ...
- 113
3
votes
0
answers
114
views
Euclidean and geodesic distance have different gradients. Does mixing the two concepts impair triplet learning?
The triplet loss is defined by Florian Schroff, Dmitry Kalenichenko, James Philbin in "FaceNet: A Unified Embedding for Face Recognition and Clustering" as
$$
\mathcal L = \sum_\mathcal T \...
- 89k
0
votes
0
answers
28
views
Optimization problem involving VAR
I'm looking to solve, from the paper https://arxiv.org/pdf/2210.02176.pdf,
$$\min_{g\in VAR(p)} \sum_{z\in \mathcal{Z}} (f( h_{x}(z))-g(z))^2\pi_{x}(z)$$
where $\mathcal{Z}\in \\{0,1\\}^{N\times W}$ ...
- 507
0
votes
0
answers
6
views
Using validation data in optimization scenarios e.g. with genetic algorithms
I'm not too familiar with optimization algorithms e.g. genetic algorithms and I'm wondering whether it makes sense to employ a validation set in this context similarly to when we train a supervised ...
- 577
2
votes
1
answer
36
views
Censored Regression model
I am considering the following censored regression model. I have a question about my R code.
Then I obtain likelihood function as follow.
Then i obtain log-likelihood function.
...
- 31
2
votes
2
answers
67
views
How do I obtain the primal and dual for the estimator $\min _\beta\left[\|\beta\|^2+\sum_{i=1}^n \xi_i^2\right]$ s.t. $\xi_i=y_i-h(x_i)^\top \beta$?
I am working on a statistical learning exercise that requires some knowledge of convex optimization which I am unfortunately lacking.
Consider the linear regression model
$$y_i=h(x_i)^\top\beta+\...
- 41
-1
votes
2
answers
78
views
Example where the initial random state of a logistic regression matters?
I am looking into how random initialisation of a model would impact final results after tuning. This is a well known problem for deep learning (NN or gbdt), notably with random initialisation and ...
- 1,544
0
votes
0
answers
13
views
Co-occurrence matrix factorization ALS
Let's say I have an item-item co-occurrence matrix that I want to factorize. I'll thus only have item factors. Is it possible to learn the factors using ALS ?
I can't see how given that we'd have to ...
- 1
0
votes
0
answers
17
views
Divide weight decay values by the learning rate values in a grid search?
I have come across a paper where the authors do a grid search over hyperparameters. In particular, they tested different learning rates and weight decays. One thing that caught my attention was that ...
0
votes
0
answers
6
views
Finding combination of bacteria that represents maximal functions encoded by them
I have the following data where rows represent functions and columns represent bacteria. I have 18 bacteria with a total of 160 functions (all funcitons are present in at least one bacteria). I want ...
- 71
1
vote
0
answers
28
views
Why I get spikes during training with vanilla gradient descent? [closed]
I developed my own NN toolbox, and it seems it works fine.
But I am not sure why I get these spikes in my loss during training:
I a training for a classification task of 2 inputs and 2 classes, ...
- 119
4
votes
2
answers
58
views
Is optimization without function evaluations (and with only gradient evaluations) possible?
I am implementing an unconstrained optimization algorithm using gradient descent. I am evaluating a cost function at a given point, evaluating the gradient at this point, and selecting the next ...
0
votes
0
answers
13
views
Measure Total Information About a System
Say I have a correlation matrix that measures how well certain components in a system track each other along some meaningful dimension, like condition state:
So, for example, component 1 and ...
- 199
0
votes
1
answer
12
views
How to speed up aftreg estimation?
I am performing an Accelerated Failure Time (AFT) regression data on a panel setting (multiple records per observation with time-dependent variables) using the eha ...
- 135
0
votes
0
answers
30
views
Learning over non-independent joint distributions
For integer $n\geq 1$, I have a "goodness" function $f_n(F)$ that takes as input a given joint CDF $F$ of $n$ variables, and spits out a number in $[0,1]$ on how "good" $F$ is. The ...
- 253
0
votes
1
answer
38
views
Cross validation and hyperparameters tuning [duplicate]
I have few questions concerning the selection of hyperparameters for predictions in cross validation.
If I understand well, during the CV, you just create folds (inner and outer for a nested CV), and ...
- 13
1
vote
0
answers
18
views
Optimizing Algorithmic Trading Parameters: A Quest for Maximizing Expected Value
I am engaged in algorithmic trading, employing specialized models that utilize various parameters to signal trading opportunities. Once I get a signal, I could execute a trade and everything is ...
- 11