Questions tagged [optimization]

Use this tag for any use of optimization within statistics.

Filter by
Sorted by
Tagged with
0 votes
0 answers
34 views

multiple testing and p-hacking when choosing from many models

I'm trying to train a random forest model with a particle swarm optimization algorithm because my target function is not smooth and unknown to me, that is, in essence, this is search and training at ...
mr.T's user avatar
  • 259
0 votes
0 answers
11 views

Is there room for finding a more efficient hybrid optimization problem, in the context of optimization algorithms for MLE?

Recently finished my statistical modelling class, but it only briefly touched on Maximum Likelihood Estimates and I thought it was an interesting topic, so I decided to go deeper in my own time. I ...
Kevin's user avatar
  • 1
0 votes
0 answers
12 views

Expectation over cost-normalized Expected improvements

Are the following two expressions equivalent if we assume the independence of f(x) and C(x)? $$ E\left[\frac{E\left[\max\left(f(x) - f(x^*), 0\right)\right]} {C(x)}\right] $$ $$ \frac{E\left[\max\...
Ridwan Salahuddeen's user avatar
2 votes
0 answers
32 views

Posterior approximation following optimization methods

I'm trying to quantify the uncertainty in a high dimensional, and multimodal posterior space. We do not have a analytical solution for the forward model, and the forward model could be expensive to ...
Geooo's user avatar
  • 21
0 votes
0 answers
16 views

Error term in SGD with momentum

I am reading the article "How Momentum really works" (https://distill.pub/2017/momentum/), and i am confused in one point: I am trying to derive the convergence rate for momentum from the ...
Patricio's user avatar
  • 121
1 vote
0 answers
14 views

Bayesian Optimization using randomForest surrogate model in R language is taking a very longer time to complete [closed]

I am running a Bayesian Optimization to optimize an objective function where the difference between the predicted validation set and the mean of initial output of the dataset is kept to the bearest ...
Ibrahim's user avatar
  • 11
0 votes
0 answers
39 views

How is the SVM optimization objective derived from the hinge loss function?

The hinge loss function, in the context of SVMs, is given as: $$ \mathcal{L}(\mathbf{\vec w}, b\,; \mathbf{\vec x}^{(i)}, y ^{(i)}) = \max(0, 1-y ^{(i)}(\mathbf{\vec w}\cdot \mathbf{\vec x}^{(i)} + b))...
Sagnik Taraphdar's user avatar
0 votes
0 answers
27 views

Robust or Stochastic Optimization Approach for Maximizing Profit with Limited Price Information

I am tackling a linear maximization problem where I need to select the optimal product among several options over a series of weeks, given certain constraints, in order to maximize future profit. The ...
anasse's user avatar
  • 1
4 votes
2 answers
124 views

What conditions are there on the exponent $p$ such that $\underset{\mu}{\arg\min}\left\{\mathbb E\left\vert X-\mu\right\vert^p\right\} $ must exist?

Let $X\sim F(x)$ be a (univariate) random variable defined by distribution function $F$. If the expected value exists, it is equal to $ \mathbb E[X] = \underset{\mu}{\arg\min}\left\{\mathbb E\left\...
Dave's user avatar
  • 62.5k
0 votes
0 answers
11 views

Hyperparameter optimization for CNN

I have a database of defect images on materials, like holes, cuts, and so on. There is not so much information inside the images, I am aware of it. I am using a CNN, in particular a ResNet50. I know ...
Jonny_92's user avatar
0 votes
0 answers
11 views

Using R to select individual resources that optimizes group effects to meet certain thresholds [closed]

See the following for a list of all constraints and references. Using R code, I would like to create an optimization that selects people (“A”,”B”,”C”) in order to maximize the number of products where ...
user avatar
-1 votes
0 answers
20 views

Why isn't ROC maximum for threshold 0.5?

the way I understand logistic regression, threshold = 0.5 basically produces a hyperplane to classify inputs which minimizes log loss (all of which is converted into a 0 to 1 range using sigmoid), so ...
SRAVAN KOTTA's user avatar
0 votes
0 answers
22 views

Scholkopf single class linear SVM equation: why ρ substracted to 1/2 ||w||² is the same as maximizing the distance

In the one class linear SVM, the equation is : $\min_{w, \rho} \frac{1}{2} \|w\|^2- \rho + C\sum_{i=1}^{n} \xi_i$ subject to: $\begin{align*} & w \cdot x_i \geq \rho - \xi_i, \\ & \xi_i \geq 0,...
Arnaud Feldmann's user avatar
0 votes
0 answers
4 views

Custom Model For Approximating Sin Function Using Backpropagation [duplicate]

I have very simple custom model which I am doing experiment with, I have model which takes one input and produce one output. the model equation is: y = sin(ax + b). (a) and (b) are single learnable ...
mohammad's user avatar
1 vote
0 answers
31 views

Closed Form Solution for MLE parameter defining Linear Combination of two multivariate normal distributions

I have one set of $n$ observations which can be described as a single vector sampled from a multivariate normal distribution of the following form: $$ (1-\lambda)\mathbb{I}_n + \lambda \Sigma_{n} $$ ...
A Friendly Fish's user avatar
0 votes
0 answers
18 views

Intercept term of logistic regression in ADMM algorithm

On page 66, the authors of article of ADMM says that the algorithm can be modified to obtain the intercept term easily in the sparse logistic regression model. Can someone explain this easy ...
mert's user avatar
  • 313
0 votes
0 answers
39 views

Huber-Loss optimisation using Stochastic Gradient Descent to estimate intercept and coefficient of regression line

What: I'm trying to minimise the Huber-Loss for a linear regression using Stochastic Gradient Descent from scratch. Problem: It seems like that the coeffcient $m$ doesn't get optimised, therefore the ...
Corbjn's user avatar
  • 111
2 votes
0 answers
71 views

Do discontinuous functions have subgradients also?

Typically, the subgradient is defined for convex functions. And convex functions are continuous. However, DeepMind's VQ-VAE paper defines a model with a discontinuous vector quantization (VQ) layer, ...
MWB's user avatar
  • 1,347
2 votes
1 answer
44 views

Computing gradient over all examples in gradient descent

I am studying about Gradient Descent and Stochastic Gradient Descent, and the text says that one of the advantages of sgd over gd is, that gd can be computationally expensive for large datasets. In ...
WalaWizon's user avatar
  • 103
11 votes
1 answer
254 views

Reconciling optimisation for log-likelihood and Brier score

Both log-likelihood and Brier score are proper scoring rules. As such, they reach the optimum when the predicted probabilities match the true ones. Since there is only one true probability for each ...
Igor F.'s user avatar
  • 9,159
1 vote
1 answer
13 views

Gambling in multiple rounds with a maximum permitted bankroll and favorable or unfavourable probabilities

This is based on a deleted question, with the premises clarified to my understanding. You are gambling in a casino with particular rules: Bets are paid off at even amounts, so if you win a round you ...
Henry's user avatar
  • 39.6k
1 vote
0 answers
22 views

Can we solve by hand the early exit multi-class classification problem? [closed]

Problem: Find a solution $\hat{\varepsilon}$ of the following minimization problem \begin{align*} &\min_{\varepsilon \in \mathbb{R}^M} \sum_{h=1}^M \varepsilon^h \hat{R}^h+\beta \sum_{h=1}^M \...
ohana's user avatar
  • 111
0 votes
0 answers
19 views

Optimal Conditional Distribution for Minimising Information-Theoretic Expression

Consider two countable sets $\mathcal{X}$ and $\mathcal{Y}$. I aim to find the conditional distribution $P_{Y|X}$ that minimizes the following expression for any $x \in \mathcal{X}$ $$\sum_y P_{Y|X}(y|...
pmoi's user avatar
  • 1
0 votes
1 answer
53 views

How to minimize a maximum of a function of 2 parameters with Pyhton [closed]

I calculate a distance between each point $c(x,y)$ and each point of $p(x,y)$. I need to find maximum among minimums of function: ...
Ivan's user avatar
  • 3
0 votes
0 answers
12 views

ML Clustering with an added condition

Problem: I want to create distance-based clusters of customers where each cluster, in sum, yields the same revenue potential. Explanation: I'm looking at thousands of customers spread throughout a ...
Tommy Lee's user avatar
0 votes
0 answers
11 views

Necessary condition for constrained optimization

Suppose $X=(X_1,\cdots,X_k)$ follows the multinomial distribution with a known size $n$ and an unknown probability vector $(p_1,\cdots,p_k)$. Find the necessary conditions for the solution to the ...
Nothing's user avatar
  • 235
2 votes
1 answer
46 views

What's the loss that is optimized in InstructGPT RL stage?

In the InstructGPT paper they define objective of RL stage as: They try to maximize this objective using PPO. I have trouble understanding how they plug this objective into the PPO though. Do they ...
Druudik's user avatar
  • 143
3 votes
1 answer
67 views

What metric should I use for a Regression model with a gamma distributed target?

Background I'm building a regression model on insurance data to predict the losses associated with a policy. I'm running an Optuna optimisation function to help me with this, but I'm struggling with ...
Connor's user avatar
  • 625
1 vote
0 answers
24 views

Interpretation of expected wealth in Kelly betting paper

This is a sub-question of another StackOverflow question. Kelly betting on horse races with uncertainty in probability estimates (Metel 2017) describes an "ECC" variant of the Kelly method ...
Reinderien's user avatar
0 votes
0 answers
6 views

References for soft constraints definition

I am looking for some reference that defines or presents in a clear way the concept of "soft constraints". The context: I writing an article. For a particular problem, I propose a solution ...
Renato Fernandes's user avatar
2 votes
1 answer
58 views

Passing a cholesky decomposition for a matrix with constrained variances to an objective function

I am trying to optimize an objective function $L(\theta)$ in which some parameters that I aim to recover belong to a covariance matrix, $\Sigma$. $\Sigma$ has a unique structure, which includes ones ...
EB727's user avatar
  • 33
3 votes
1 answer
152 views

Scenario where minimizing 0-1 loss is different than minimizing hinge loss

Suppose we're using linear predictors. I'm trying to conceptually understand how minimizing hinge loss and 0-1 loss aren't necessarily the same. For instance I was told that one can choose a set of ...
redbull_nowings's user avatar
3 votes
2 answers
59 views

Why can the method of moments be expressed as a minimization problem?

Generalized method of moments (GMM) estimation seems to be called generalized method of moments because the standard method of moments (MoM) is a special case, following the following logic. MoM is ...
Dave's user avatar
  • 62.5k
0 votes
0 answers
37 views

about bayesian optimization, please help [duplicate]

in machine learning mastery website, i don't understand this paragraph. P(A|B) = P(B|A) * P(A) / P(B) We can simplify this calculation by removing the normalizing value of P(B) and describe the ...
Kelvin Wijaya's user avatar
0 votes
0 answers
66 views

Advice on Gaussian Process Classifier optimisation best practises? [duplicate]

Hyperparameter Range Determination: My main challenge is in setting effective ranges for hyperparameters such as length_scale, noise_level, and sigma_0. Currently, for length_scale, I've used the ...
Achilleas Pavlou's user avatar
2 votes
1 answer
32 views

Training loss reach to zero, then suddenly increases, then decreases to zero

I get the following loss behavior when training multilayer perceptron with mean squared error loss on some synthetic data using default Adam with default learning. (I am working on 1 demention data) I ...
Rahim Brahimi's user avatar
2 votes
0 answers
16 views

Transforming discrete optimisation problem into continuous optimisation problem

In Sparse Hilbert-Schmidt Independence Criterion Regression (Poignard and Yamada, AISTATS 2020), the authors consider a way to perform feature selection by taking the subset of features that maximises ...
LoveRKHS's user avatar
0 votes
1 answer
32 views

Is there a (lower) limit/minimum for learning rate values?

I'm building a model for traffic prediction with ConvLSTM and A3T-GCN cells. Since the input data is highly complex and the model is relatively big, I can only load ...
olenscki's user avatar
  • 101
1 vote
1 answer
25 views

Name for adaptive simulated annealing that cyclically decreases stepsize, then increases temperature, resetting both after each accepted move?

I have a model with many discrete parameters between 1 and 99. Each step new parameters are sampled from a discrete uniform distribution with variable stepsize around the current parameter value, ...
Livid's user avatar
  • 1,158
1 vote
0 answers
28 views

Open-Source Alternative of Closed-Source Swarm Optimization

We are currently using a closed-source software for model optimization. Since it is closed, we have no transparency into the underlying code, which therefore by definition makes it challenging to ...
Oliver Angelil's user avatar
2 votes
0 answers
74 views

Fitting curves with restricted relative orientations

Edited to focus on the math (thanks whuber): Given a sequence $z_1,…,z_n$ of complex numbers and a fixed real number $σ$, find a sequence $x_1,…,x_n$ from the set $\{0,\pm1,\pm2\}$ minimizing $$ \sum_{...
CrazyArm's user avatar
  • 121
0 votes
0 answers
91 views

What's the difference and relationship between theta, theta star and theta hat?

I understand that $\theta$ is the true distribution parameter (great explanation here). I also know that $\hat\theta$ is an estimator of the true $\theta$ (so for example, MLE is an example of $\hat\...
HeyJude's user avatar
  • 350
0 votes
0 answers
34 views

Online learning with strongly convex loss w.r.t. L1 norm

Consider an online learning problem where the loss function observed by the algorithm at time $t$ can be written as $$ f_t(\lambda) = \langle V_t, w \rangle + \mu \sum \lambda_i \log \lambda_i $$ for ...
Vassily's user avatar
  • 141
3 votes
0 answers
40 views

How do I change my Scipy.Optimize.Minimize configuration to better find solution for constrained optimization? [closed]

I'm trying to do constrained optimization with Scipy.Optimize.Minimize and I keep on getting no solution, when I know there is a solution, because I can find one ...
confused's user avatar
  • 3,253
0 votes
0 answers
15 views

Parameter estimation for HMM with different emission distribution for different states

I am trying to solve a parameter estimation problem where a HMM have different emission distribution for different states (i.e. Gaussian for one state and exponential for the other). However most ...
Geng Wang's user avatar
0 votes
0 answers
32 views

Learning the gradient descent stepsize

I'm trying to apply reinforcement learning to learn the optimal stepsize in every gradient descent step. Currently I'm only considering simple quadratic problems of the form f(x)=x'Qx with Q diagonal ...
CodeGuy's user avatar
0 votes
0 answers
22 views

Learning the gradient descent stepsize

I'm trying to apply reinforcement learning to learn the optimal stepsize in every gradient descent step. Currently I'm only considering simple quadratic problems of the form f(x)=x'Qx with Q diagonal ...
CodeGuy's user avatar
0 votes
0 answers
24 views

Numerical Optimization of Marginal Likelihood that Explodes

I have a model with a marginal likelihood of the following form: $$\mathcal{L}(\theta_1, \theta_2, \theta_3|\{x_{i,j}\}_{i=1, j=1}^{N, M_i})=\prod_{i=1}^{N}\int_{0}^{1} f(p_i;\theta_1) \prod_{j=1}^{...
Drunk Deriving's user avatar
1 vote
0 answers
9 views

Issues with gradient of standard deviation in GPR using skopt.learning.gaussian_process

I'm currently working on a Gaussian Process Regression (GPR) model using the implementation provided in skopt.learning.GaussianProcessRegressor which is a wrapper for the sklearn implementation. This ...
Dave's user avatar
  • 11
1 vote
0 answers
17 views

How to compare different clusters of different size, rotation, scale and translation?

Assume that you have a matrix $X$ that contains the data inside the left image. The data inside $X$ is not classified. The matrix $X$ also contains outliers/noise. On the right, we can se the template ...
euraad's user avatar
  • 415

1
2 3 4 5
56