Skip to main content

Questions tagged [optimization]

Use this tag for any use of optimization within statistics.

Filter by
Sorted by
Tagged with
0 votes
0 answers
11 views

Derivation of dual formulation of support vector regression

I'm trying to derive the dual formulation of epsilon-insensitive support vector regression. I think my derivation is correct, but I can't match it up to a result for the dual that I've seen given in ...
oweydd's user avatar
  • 225
2 votes
1 answer
33 views

Regression with known upper bounds and lower bounds of predicted variables

I have three variables $x_1$, $x_2$ and $x_3$ to predict $y$. Simplest regression setup is to run regression $y \sim x_1 + x_2 + x_3$. Then I have prediction $\hat{y} = \hat \beta_1 x_1 + \hat \...
Matt Frank's user avatar
0 votes
0 answers
22 views

minimization involved $l_2$ norm [migrated]

I am trying to find the minimum of the following problem $$\frac{\theta}{2}\lVert\beta-x\rVert_2^2+\lambda\lVert x\rVert_2-\frac{1}{2\tau}\lVert x\rVert_2^2+\alpha^Tx$$ by taking the derivative with ...
Simple's user avatar
  • 207
1 vote
0 answers
17 views

SVRG vs full gradient descent

Stochastic gradient descent allows us to avoid the computation of full gradients at the expense of introducing a noise floor to convergence. To decrease this noise floor, SGD requires a decrease in ...
hegash's user avatar
  • 111
2 votes
1 answer
71 views

Sampling to maximise f(x)p(x)

I have a probability distribution $p(x)$ that I can generate samples form really easily. I also have some function $f(x)$ that I can calculate for each sample. My goal is to estimate the value of $x$ ...
DBruwel's user avatar
  • 43
0 votes
0 answers
14 views

Manual MLE of AR(1) yields a weird initial value $y_0$

I am playing with a manual implementation of the maximum likelihood estimator (MLE) of the parameters in an AR(1) model $$ y_t = c + \varphi_1 y_{t-1} + \varepsilon_t $$ with $\text{Var}(\varepsilon_t)...
Richard Hardy's user avatar
0 votes
0 answers
7 views

How to find a linear decision boundary of a linearly separable problem with unlimited class evaluations?

I have a binary classification problem, where my goal is to find a linear decision boundary (which I assume exists). The context of the problem is that I have an iterative optimization process, where ...
oskar0711's user avatar
0 votes
0 answers
15 views

Optimisation of Polynomial Fittting Process

I have built a multitvariate log link GLM model and I want to fit polynomials to some of the numerical variates (i.e. fit polynomials of order 1,2,3 etc to the relativities of the model). However, I ...
JHARR's user avatar
  • 1
1 vote
1 answer
72 views

Non-linear regression with very noisy data with nls() in R

I am trying to fit noisy data to a specific model with two parameters which I would like to estimate. Unfortunately, the model fit is just terrible with added noise. Is there anything I can do to ...
leze's user avatar
  • 11
1 vote
0 answers
33 views

Optimizing objective with two variables multiplied with each other? [closed]

Let's say you want to optimize the following objective function: $$\min_{a,b} \Vert a + ab + b - W \Vert_2^2$$ where $a \in \mathbb{R}^{m,n}$ and $b \in \mathbb{R}^{n,p}$ are the learnable matrices, ...
user3667125's user avatar
0 votes
0 answers
18 views

Adam's $\beta_1$ fixed in practice but required to depend on $t$ for convergence proofs

In the paper ADAM: A METHOD FOR STOCHASTIC OPTIMIZATION, the exponential moving average parameter $\beta_1$ is set to $0.9$ as default in most ML/DL APIs but the convergence proof requires that $\...
Toonia's user avatar
  • 13
1 vote
0 answers
33 views

Fitting a model with multiple inputs, multiple outputs, multiple parameters, and covariance matrices for each data point

This question is the theoretical counterpart to another question posted on StackOverflow, where I asked about the implementation of the fitting algorithm using Scipy or lmfit libraries for Python. ...
Swike's user avatar
  • 139
1 vote
0 answers
31 views

Fitting the rotation between two sets of 3D points, given 1D measurements

Context: I am measuring a series of points on the surface of an object, with a measuring device which can only capture the position of a point perpendicular to the the surface being measured. I am ...
rr-mark's user avatar
  • 11
0 votes
0 answers
13 views

Connection between mean update in CMA-ES and gradient of expected fitness

I currently learn about black-box optimization and CMA-ES. Now, I try to understand some of the theoretical foundations of it. The update of the mean in classic CMA-ES is as follows: $$m \leftarrow m +...
HansDoe's user avatar
0 votes
0 answers
11 views

Is there room for finding a more efficient hybrid optimization problem, in the context of optimization algorithms for MLE?

Recently finished my statistical modelling class, but it only briefly touched on Maximum Likelihood Estimates and I thought it was an interesting topic, so I decided to go deeper in my own time. I ...
Kevin's user avatar
  • 1
0 votes
0 answers
12 views

Expectation over cost-normalized Expected improvements

Are the following two expressions equivalent if we assume the independence of f(x) and C(x)? $$ E\left[\frac{E\left[\max\left(f(x) - f(x^*), 0\right)\right]} {C(x)}\right] $$ $$ \frac{E\left[\max\...
Ridwan Salahuddeen's user avatar
2 votes
0 answers
37 views

Posterior approximation following optimization methods

I'm trying to quantify the uncertainty in a high dimensional, and multimodal posterior space. We do not have a analytical solution for the forward model, and the forward model could be expensive to ...
Geooo's user avatar
  • 21
0 votes
0 answers
17 views

Error term in SGD with momentum

I am reading the article "How Momentum really works" (https://distill.pub/2017/momentum/), and i am confused in one point: I am trying to derive the convergence rate for momentum from the ...
Patricio's user avatar
  • 121
1 vote
0 answers
19 views

Bayesian Optimization using randomForest surrogate model in R language is taking a very longer time to complete [closed]

I am running a Bayesian Optimization to optimize an objective function where the difference between the predicted validation set and the mean of initial output of the dataset is kept to the bearest ...
Ibrahim's user avatar
  • 11
0 votes
0 answers
41 views

How is the SVM optimization objective derived from the hinge loss function?

The hinge loss function, in the context of SVMs, is given as: $$ \mathcal{L}(\mathbf{\vec w}, b\,; \mathbf{\vec x}^{(i)}, y ^{(i)}) = \max(0, 1-y ^{(i)}(\mathbf{\vec w}\cdot \mathbf{\vec x}^{(i)} + b))...
Sagnik Taraphdar's user avatar
0 votes
0 answers
28 views

Robust or Stochastic Optimization Approach for Maximizing Profit with Limited Price Information

I am tackling a linear maximization problem where I need to select the optimal product among several options over a series of weeks, given certain constraints, in order to maximize future profit. The ...
anasse's user avatar
  • 11
4 votes
2 answers
134 views

What conditions are there on the exponent $p$ such that $\underset{\mu}{\arg\min}\left\{\mathbb E\left\vert X-\mu\right\vert^p\right\} $ must exist?

Let $X\sim F(x)$ be a (univariate) random variable defined by distribution function $F$. If the expected value exists, it is equal to $ \mathbb E[X] = \underset{\mu}{\arg\min}\left\{\mathbb E\left\...
Dave's user avatar
  • 64.2k
0 votes
0 answers
11 views

Hyperparameter optimization for CNN

I have a database of defect images on materials, like holes, cuts, and so on. There is not so much information inside the images, I am aware of it. I am using a CNN, in particular a ResNet50. I know ...
Jonny_92's user avatar
0 votes
0 answers
24 views

Scholkopf single class linear SVM equation: why ρ substracted to 1/2 ||w||² is the same as maximizing the distance

In the one class linear SVM, the equation is : $\min_{w, \rho} \frac{1}{2} \|w\|^2- \rho + C\sum_{i=1}^{n} \xi_i$ subject to: $\begin{align*} & w \cdot x_i \geq \rho - \xi_i, \\ & \xi_i \geq 0,...
Arnaud Feldmann's user avatar
0 votes
0 answers
4 views

Custom Model For Approximating Sin Function Using Backpropagation [duplicate]

I have very simple custom model which I am doing experiment with, I have model which takes one input and produce one output. the model equation is: y = sin(ax + b). (a) and (b) are single learnable ...
mohammad's user avatar
1 vote
0 answers
34 views

Closed Form Solution for MLE parameter defining Linear Combination of two multivariate normal distributions

I have one set of $n$ observations which can be described as a single vector sampled from a multivariate normal distribution of the following form: $$ (1-\lambda)\mathbb{I}_n + \lambda \Sigma_{n} $$ ...
A Friendly Fish's user avatar
0 votes
0 answers
18 views

Intercept term of logistic regression in ADMM algorithm

On page 66, the authors of article of ADMM says that the algorithm can be modified to obtain the intercept term easily in the sparse logistic regression model. Can someone explain this easy ...
mert's user avatar
  • 313
0 votes
0 answers
59 views

Huber-Loss optimisation using Stochastic Gradient Descent to estimate intercept and coefficient of regression line

What: I'm trying to minimise the Huber-Loss for a linear regression using Stochastic Gradient Descent from scratch. Problem: It seems like that the coeffcient $m$ doesn't get optimised, therefore the ...
Corbjn's user avatar
  • 111
2 votes
0 answers
72 views

Do discontinuous functions have subgradients also?

Typically, the subgradient is defined for convex functions. And convex functions are continuous. However, DeepMind's VQ-VAE paper defines a model with a discontinuous vector quantization (VQ) layer, ...
MWB's user avatar
  • 1,337
2 votes
1 answer
46 views

Computing gradient over all examples in gradient descent

I am studying about Gradient Descent and Stochastic Gradient Descent, and the text says that one of the advantages of sgd over gd is, that gd can be computationally expensive for large datasets. In ...
WalaWizon's user avatar
  • 103
11 votes
1 answer
272 views

Reconciling optimisation for log-likelihood and Brier score

Both log-likelihood and Brier score are proper scoring rules. As such, they reach the optimum when the predicted probabilities match the true ones. Since there is only one true probability for each ...
Igor F.'s user avatar
  • 9,353
1 vote
1 answer
16 views

Gambling in multiple rounds with a maximum permitted bankroll and favorable or unfavourable probabilities

This is based on a deleted question, with the premises clarified to my understanding. You are gambling in a casino with particular rules: Bets are paid off at even amounts, so if you win a round you ...
Henry's user avatar
  • 40.2k
1 vote
0 answers
22 views

Can we solve by hand the early exit multi-class classification problem? [closed]

Problem: Find a solution $\hat{\varepsilon}$ of the following minimization problem \begin{align*} &\min_{\varepsilon \in \mathbb{R}^M} \sum_{h=1}^M \varepsilon^h \hat{R}^h+\beta \sum_{h=1}^M \...
ohana's user avatar
  • 111
0 votes
0 answers
21 views

Optimal Conditional Distribution for Minimising Information-Theoretic Expression

Consider two countable sets $\mathcal{X}$ and $\mathcal{Y}$. I aim to find the conditional distribution $P_{Y|X}$ that minimizes the following expression for any $x \in \mathcal{X}$ $$\sum_y P_{Y|X}(y|...
pmoi's user avatar
  • 1
0 votes
1 answer
53 views

How to minimize a maximum of a function of 2 parameters with Pyhton [closed]

I calculate a distance between each point $c(x,y)$ and each point of $p(x,y)$. I need to find maximum among minimums of function: ...
Ivan's user avatar
  • 3
0 votes
0 answers
12 views

ML Clustering with an added condition

Problem: I want to create distance-based clusters of customers where each cluster, in sum, yields the same revenue potential. Explanation: I'm looking at thousands of customers spread throughout a ...
Tommy Lee's user avatar
0 votes
0 answers
11 views

Necessary condition for constrained optimization

Suppose $X=(X_1,\cdots,X_k)$ follows the multinomial distribution with a known size $n$ and an unknown probability vector $(p_1,\cdots,p_k)$. Find the necessary conditions for the solution to the ...
Nothing's user avatar
  • 287
2 votes
1 answer
61 views

What's the loss that is optimized in InstructGPT RL stage?

In the InstructGPT paper they define objective of RL stage as: They try to maximize this objective using PPO. I have trouble understanding how they plug this objective into the PPO though. Do they ...
Druudik's user avatar
  • 143
3 votes
1 answer
76 views

What metric should I use for a Regression model with a gamma distributed target?

Background I'm building a regression model on insurance data to predict the losses associated with a policy. I'm running an Optuna optimisation function to help me with this, but I'm struggling with ...
Connor's user avatar
  • 635
1 vote
0 answers
25 views

Interpretation of expected wealth in Kelly betting paper

This is a sub-question of another StackOverflow question. Kelly betting on horse races with uncertainty in probability estimates (Metel 2017) describes an "ECC" variant of the Kelly method ...
Reinderien's user avatar
0 votes
0 answers
7 views

References for soft constraints definition

I am looking for some reference that defines or presents in a clear way the concept of "soft constraints". The context: I writing an article. For a particular problem, I propose a solution ...
Renato Fernandes's user avatar
2 votes
1 answer
65 views

Passing a cholesky decomposition for a matrix with constrained variances to an objective function

I am trying to optimize an objective function $L(\theta)$ in which some parameters that I aim to recover belong to a covariance matrix, $\Sigma$. $\Sigma$ has a unique structure, which includes ones ...
EB727's user avatar
  • 33
3 votes
1 answer
166 views

Scenario where minimizing 0-1 loss is different than minimizing hinge loss

Suppose we're using linear predictors. I'm trying to conceptually understand how minimizing hinge loss and 0-1 loss aren't necessarily the same. For instance I was told that one can choose a set of ...
redbull_nowings's user avatar
3 votes
2 answers
68 views

Why can the method of moments be expressed as a minimization problem?

Generalized method of moments (GMM) estimation seems to be called generalized method of moments because the standard method of moments (MoM) is a special case, following the following logic. MoM is ...
Dave's user avatar
  • 64.2k
0 votes
0 answers
37 views

about bayesian optimization, please help [duplicate]

in machine learning mastery website, i don't understand this paragraph. P(A|B) = P(B|A) * P(A) / P(B) We can simplify this calculation by removing the normalizing value of P(B) and describe the ...
Kelvin Wijaya's user avatar
0 votes
0 answers
66 views

Advice on Gaussian Process Classifier optimisation best practises? [duplicate]

Hyperparameter Range Determination: My main challenge is in setting effective ranges for hyperparameters such as length_scale, noise_level, and sigma_0. Currently, for length_scale, I've used the ...
Achilleas Pavlou's user avatar
2 votes
1 answer
34 views

Training loss reach to zero, then suddenly increases, then decreases to zero

I get the following loss behavior when training multilayer perceptron with mean squared error loss on some synthetic data using default Adam with default learning. (I am working on 1 demention data) I ...
Rahim Brahimi's user avatar
2 votes
0 answers
23 views

Transforming discrete optimisation problem into continuous optimisation problem

In Sparse Hilbert-Schmidt Independence Criterion Regression (Poignard and Yamada, AISTATS 2020), the authors consider a way to perform feature selection by taking the subset of features that maximises ...
LoveRKHS's user avatar
0 votes
1 answer
34 views

Is there a (lower) limit/minimum for learning rate values?

I'm building a model for traffic prediction with ConvLSTM and A3T-GCN cells. Since the input data is highly complex and the model is relatively big, I can only load ...
olenscki's user avatar
  • 101
1 vote
1 answer
27 views

Name for adaptive simulated annealing that cyclically decreases stepsize, then increases temperature, resetting both after each accepted move?

I have a model with many discrete parameters between 1 and 99. Each step new parameters are sampled from a discrete uniform distribution with variable stepsize around the current parameter value, ...
Livid's user avatar
  • 1,168

1
2 3 4 5
56