Questions tagged [optimization]
Use this tag for any use of optimization within statistics.
2,800
questions
0
votes
0
answers
10
views
Recursive Random Search and Categorical Cost Functions
I'm currently working on a project that involves optimizing the default Spark-submit configurations to minimize execution time. I've developed two models to aid in this process:
Binary Classification ...
- 155
1
vote
0
answers
22
views
Some further explanation of Alex Smola's 1998 implementation of support vector regression
I am currently going through, and trying to implement the pseudo-code in Alex Smola's 1998 paper on support vector regression, particularly the one on sequential minimal optimization. (Section 4.6.3, ...
- 11
0
votes
0
answers
19
views
Best betting strategy with positive EV while avoiding large loss
Coming from a financial stop loss background. Let's say in a game of $T$ rounds:
You start with $X_0=100$. And your profit $P=0$.
At round $t$, you can choose the bet size $z_t$. You will get $z_t\...
- 811
1
vote
1
answer
50
views
Simple problem of optimizing a positive definite matrix
I am working on writing a simple tutorial about constrained optimization. I plan to use two examples: constraining a vector to have unit norm, and constraining a matrix to be symmetric positive ...
- 1,320
0
votes
0
answers
41
views
Can IRLS deal with inequallity constrains? What should I use otherwise?
I have a set of observed data points $p_i = (a_i,b_i)$, and a common constant $c$. Theoretically, the points are supposed to follow the equation:
$$a_i = b_i + x_1 + x_2(b_i + c)$$
I aim to find the ...
- 73
3
votes
0
answers
29
views
Is it more efficient to optimize precision than covariance matrix?
This might be a silly question, but I want to make sure I'm not missing something.
Say that we want to fit a multivariate Gaussian distribution $\mathcal{N}(\mu, \Sigma)$ to some data by maximizing ...
- 1,320
0
votes
0
answers
21
views
Are interior-point methods guranteed to converge to the global optimum of a convex objective function?
I am looking into convex optimization. However, I am not sure if there are interior-point methods that are guaranteed to converge to the globally optimal solution given either a strictly convex or a ...
- 101
1
vote
1
answer
34
views
Why can a classifier's predicted labels be improved (with respect to the same metric the classifier optimizes) by adjusting classification threshold?
I am hoping to enhance my (and others' perhaps too) understanding of some basic principles, which seem surprisingly elusive.
For a start, I would like to consider imbalanced binary classification, ...
- 11
0
votes
0
answers
37
views
Maximum Likelihood Estimation with Gradient Descent and Squarred Loss
My goal is to learn parameters $\mu$ and $\sigma$ of a univariate Gaussian distribution using gradient descent to validate my understanding of the algorithm by deriving all the formulas from scratch. ...
- 1
0
votes
0
answers
27
views
Should the target be standardized in gradient descent?
Suppose that we have a general loss function that depends on some parameters $w$ (e.g. neural network weights):
$$L_w =\frac{1}{N} \sum_i \ell(\hat{y}_i, y_i)$$
Is it beneficial to standardize the ...
- 541
0
votes
0
answers
13
views
Trying to callibrate SIRD model to real data using least squares optimisation
I am trying to fit SIRD model in R to real data. However, the observed values are lying nowhere on the fitted curve. I can't understand what the error is or how to resolve it. My data is the Mexican ...
1
vote
0
answers
9
views
Two variants of Nesterov Accelerated Gradient: are they equivalent?
I was puzzled to find that the description of the Nesterov Accelerated Gradient on Paperswithcode, namely:
$v_t = \beta * v_{t-1} \color{red}{+} \eta * ∇ J(\theta \color{red}{-} \beta * v_{t-1})$
$\...
0
votes
0
answers
25
views
Why is Stochastic Gradient Descent valid?
It seems unclear to me why SGD/minibatch GD works. I heard from someone that SGD works because "as commonly seen in stochastic optimization, the gradient step is an unbiased estimator of the true ...
- 173
3
votes
1
answer
67
views
Can I use a likelihood-ratio test when the measure of deviance between two models is not the log-likelihood?
We use the Nelder-Mead optimziation algorithm (as implemented in the dfoptim package for R) to fit a model with several free parameters.
What is minimised (in our current implementation) when ...
- 75
0
votes
0
answers
13
views
HMMs "difficulty" compared to a Markov model
Given an HMM, it is easy to compute the best approximating $n$-gram model over the observations. For example, for $N=1$, we have $p(w_i|w_{i-1}) = \sum_{s_i,s_{i-1}}p(w_i,s_i|w_{i-1},s_{i-1})=\sum_{...
- 277
0
votes
0
answers
24
views
What kind of classifiers we shouldn't use for feature selection?
Generally, I see that, for feature-selection, people use PSO as optimizer and inside the cost function, they use less powerful classifiers like SVC, Logistic regression, KNN, etc.
Is there a reason ...
- 1,815
0
votes
0
answers
42
views
Ways to parametrise a positive parameter
I am working with a differentiable state-space model involving a noise variance term $\sigma^2$ which I want to parametrise based on some features, e.g. $\sigma^2 = g(X\beta) > 0$, wherer $\beta$ ...
- 121
4
votes
2
answers
77
views
Confusion over Fisher-scoring algorithm
Given a probability model $f(X;\theta)$ and a set of i.i.d. observations $x_1,\ldots,x_n$ which we assume to be drawn from some true parameter $f(X; \theta_0)$, we can perform maximum-likelihood ...
- 316
1
vote
0
answers
31
views
Ask a coding problem for the equivalence of unconstrained Optimization with L1 Regularization
I recently read a statistics paper: DAGs with NO TEARS: Continuous Optimization for Structure Learning
It has an unconstrained problem:
$$\min_\theta F(\theta)+\lambda || \theta||_1$$, where $$F(\...
- 21
0
votes
0
answers
11
views
Inverse Problem: Using LightGBM model to recommend X (feature) ranges to achieve a specific y (target) range
I am trying to build a LightGBM regression model, where in I have aroud 15-20 Input features and my target variable within a range of 20-40.
I have used the SHAP beeswarm plot to kind of understand ...
- 121
1
vote
0
answers
19
views
Ask a coding problem for the equivalence of unconstrained Optimization with L1 Regularization [duplicate]
I recently read a statistics paper. It has an unconstrained problem:
$$\min_\theta F(\theta)+\lambda || \theta||_1$$, where $$F(\theta)=L(\theta)+\frac{\rho}{2}|h(W(\theta))|^2+ \alpha h(W(\theta))$$
$...
- 21
2
votes
1
answer
36
views
Do convergence rates for (convex) gradient descent apply when domain is (convex) subset of reals?
I have a convex multi-variate optimization problem where each variable lies on the domain $[x, \infty)$ for some positive number $x$. I know the problem has a unique finite solution in the domain, ...
- 133
0
votes
0
answers
16
views
Optimize two functions on two datasets with shared parameters
I have two functions that share parameters and each function needs to be optimized on separate data. My question is: can I simply add the residual sum of squares for two functions in my objective ...
- 131
1
vote
0
answers
24
views
Optimization fails to converge on known parameters for zero-inflated beta binomial distribution [closed]
I am trying to fit to simulated zero-inflated beta binomial data using the distributions provided by VGAM in R.
When using optim on a likelihood function I wrote, ...
- 11
0
votes
0
answers
33
views
Top-N recommender system
Say an intermediary is using a two part recommender model that attempts to facilitate services between its clients and external vendors:
Model 1: Predict probability of vendor bidding on a given ...
0
votes
0
answers
13
views
Nonlinear Optimization of Noisy Functions w/ Bound Constraints via SciPy
Can we use scipy.optimize.minimize to find the best parameters $\mathbf{w} \in \Omega^k$,
$\Omega \subset \mathbb{R}$, of a function
$g = g(f(\mathbf{x}), \mathbf{w}...
- 304
3
votes
1
answer
112
views
Kalman Filter to minimize weighted errors on the states: what's wrong with my derivation
I am thinking about how to implement a "weighted Kalman Filter". Note that the weights here are on the states. Basically the classical KF minimizes $\sum (x_i - \hat{x_i} )^2$ but I want to ...
- 230
2
votes
1
answer
133
views
Closed Form Solution for Gaussian Matrix which is Convex Combination?
I already asked a pretty similar question here, but the answer was inconclusive and now this related problem has come up again here.
My problem is as follows, I have a $2n$-dimensional multivariate ...
- 425
0
votes
0
answers
16
views
Why a project a reshape to 4x4x1024 for DCGAN?
In the paper Unsupervised Representation Learning with Deep Convolution Generative Adversarial Networks by Radford et. al. (2015), the model described projects and reshapes a 100 valued noise vector ...
0
votes
0
answers
13
views
Fitting a policy to a target distribution $\pi$ with projections
Given a discrete target distribution $\mathbf{\pi}\in\Delta^n$, fitting a policy $\mathbf{p}$ to this distribution can be done via cross entropy loss, that is, minimizing $-\pi^\top \log \mathbf{p}$.
...
- 205
1
vote
0
answers
21
views
Derivation of dual formulation of support vector regression
I'm trying to derive the dual formulation of epsilon-insensitive support vector regression. I think my derivation is correct, but I can't match it up to a result for the dual that I've seen given in ...
- 225
2
votes
1
answer
81
views
Regression with known upper bounds and lower bounds of predicted variables
I have three variables $x_1$, $x_2$ and $x_3$ to predict $y$. Simplest regression setup is to run regression $y \sim x_1 + x_2 + x_3$. Then I have prediction $\hat{y} = \hat \beta_1 x_1 + \hat \...
- 230
1
vote
0
answers
20
views
SVRG vs full gradient descent
Stochastic gradient descent allows us to avoid the computation of full gradients at the expense of introducing a noise floor to convergence. To decrease this noise floor, SGD requires a decrease in ...
- 111
2
votes
1
answer
72
views
Sampling to maximise f(x)p(x)
I have a probability distribution $p(x)$ that I can generate samples form really easily. I also have some function $f(x)$ that I can calculate for each sample. My goal is to estimate the value of $x$ ...
- 43
1
vote
0
answers
39
views
Manual MLE of AR(1) yields a weird initial value $y_0$
I am playing with a manual implementation of the maximum likelihood estimator (MLE) of the parameters in an AR(1) model
$$
y_t = c + \varphi_1 y_{t-1} + \varepsilon_t
$$
with $\text{Var}(\varepsilon_t)...
- 68.8k
0
votes
0
answers
9
views
How to find a linear decision boundary of a linearly separable problem with unlimited class evaluations?
I have a binary classification problem, where my goal is to find a linear decision boundary (which I assume exists). The context of the problem is that I have an iterative optimization process, where ...
0
votes
0
answers
20
views
Optimisation of Polynomial Fittting Process
I have built a multitvariate log link GLM model and I want to fit polynomials to some of the numerical variates (i.e. fit polynomials of order 1,2,3 etc to the relativities of the model). However, I ...
- 1
1
vote
1
answer
105
views
Non-linear regression with very noisy data with nls() in R
I am trying to fit noisy data to a specific model with two parameters which I would like to estimate. Unfortunately, the model fit is just terrible with added noise. Is there anything I can do to ...
- 11
1
vote
0
answers
36
views
Optimizing objective with two variables multiplied with each other? [closed]
Let's say you want to optimize the following objective function:
$$\min_{a,b} \Vert a + ab + b - W \Vert_2^2$$
where $a \in \mathbb{R}^{m,n}$ and $b \in \mathbb{R}^{n,p}$ are the learnable matrices, ...
- 121
0
votes
0
answers
18
views
Adam's $\beta_1$ fixed in practice but required to depend on $t$ for convergence proofs
In the paper ADAM: A METHOD FOR STOCHASTIC OPTIMIZATION, the exponential moving average parameter $\beta_1$ is set to $0.9$ as default in most ML/DL APIs but the convergence proof requires that $\...
- 13
1
vote
0
answers
34
views
Fitting a model with multiple inputs, multiple outputs, multiple parameters, and covariance matrices for each data point
This question is the theoretical counterpart to another question posted on StackOverflow, where I asked about the implementation of the fitting algorithm using Scipy or lmfit libraries for Python. ...
- 139
1
vote
0
answers
33
views
Fitting the rotation between two sets of 3D points, given 1D measurements
Context:
I am measuring a series of points on the surface of an object, with a measuring device which can only capture the position of a point perpendicular to the the surface being measured.
I am ...
- 11
0
votes
0
answers
22
views
Connection between mean update in CMA-ES and gradient of expected fitness
I currently learn about black-box optimization and CMA-ES. Now, I try to understand some of the theoretical foundations of it. The update of the mean in classic CMA-ES is as follows:
$$m \leftarrow m +...
- 1
0
votes
0
answers
11
views
Is there room for finding a more efficient hybrid optimization problem, in the context of optimization algorithms for MLE?
Recently finished my statistical modelling class, but it only briefly touched on Maximum Likelihood Estimates and I thought it was an interesting topic, so I decided to go deeper in my own time. I ...
- 1
0
votes
0
answers
13
views
Expectation over cost-normalized Expected improvements
Are the following two expressions equivalent if we assume the independence of f(x) and C(x)?
$$
E\left[\frac{E\left[\max\left(f(x) - f(x^*), 0\right)\right]} {C(x)}\right]
$$
$$
\frac{E\left[\max\...
2
votes
0
answers
38
views
Posterior approximation following optimization methods
I'm trying to quantify the uncertainty in a high dimensional, and multimodal posterior space. We do not have a analytical solution for the forward model, and the forward model could be expensive to ...
- 21
0
votes
0
answers
17
views
Error term in SGD with momentum
I am reading the article "How Momentum really works" (https://distill.pub/2017/momentum/), and i am confused in one point:
I am trying to derive the convergence rate for momentum from the ...
- 121
1
vote
0
answers
46
views
Bayesian Optimization using randomForest surrogate model in R language is taking a very longer time to complete [closed]
I am running a Bayesian Optimization to optimize an objective function where the difference between the predicted validation set and the mean of initial output of the dataset is kept to the bearest ...
- 11
0
votes
0
answers
42
views
How is the SVM optimization objective derived from the hinge loss function?
The hinge loss function, in the context of SVMs, is given as:
$$
\mathcal{L}(\mathbf{\vec w}, b\,; \mathbf{\vec x}^{(i)}, y ^{(i)}) = \max(0, 1-y ^{(i)}(\mathbf{\vec w}\cdot \mathbf{\vec x}^{(i)} + b))...
- 115
0
votes
0
answers
29
views
Robust or Stochastic Optimization Approach for Maximizing Profit with Limited Price Information
I am tackling a linear maximization problem where I need to select the optimal product among several options over a series of weeks, given certain constraints, in order to maximize future profit. The ...
- 11