Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [optimization]

Use this tag for any use of optimization within statistics.

0
votes
0answers
4 views

Stochastic gradient variance reduced methods

I'm doing stochastic gradient descent on a non-convex optimization problem. Gradient corresponds to an intractable expectation which I approximate via Monte Carlo averaging. I'm trying to infer the ...
0
votes
0answers
5 views

Fitting Gamma Distribution in Python with scalar factor

I have done optimization to minimize error of resulted model, however, the R-squared is still low. I try to multiply a scalar to the resulted model to obtain a better fit. The scalar will be generated ...
0
votes
0answers
9 views

Covariance of an estimate from optimization

Consider a standard linear regression model, $\boldsymbol y = X \boldsymbol \beta + \boldsymbol \epsilon$. $\boldsymbol y$ is a vector of $m$ responses, $X$ is a design matrix with $m$ rows and $p$ ...
0
votes
0answers
11 views

Linear Programming/optimisation with R? [on hold]

I hope this isn't a repeated question, but: does anybody have a firm idea of how much data can be handled by R when doing a linear optimisation model? For example, will R be able to handle ...
0
votes
1answer
28 views

Singular gradient erros, NLS in R

I'm trying to fit nls(Mound~ a*kg.bag.collar^b + c, start = list(a = 83, b = -.5, c=100), data=test) using the dataset here. I've fit it without trouble without the ...
0
votes
0answers
19 views

How can I concentrate out parameters entering linearly in a partially non-linear regression?

I have model $$y_i = x_i^\top\beta + \delta \exp(w_i\eta) + \epsilon_i$$ in setting up a non-linear regression problem $$\min_{\beta,\delta,\eta} \frac{1}{2N} \sum_i^N (y_i - x_i^\top\beta - \delta \...
0
votes
0answers
24 views

An optimal stochastic problem and Monte-carlo

Assume we are in a Brownian filtration where I denote $W$ the Brownian motion. My problem is to numerically compute $$ \min_X E (\int^1_0 X^2_tdt),\ \ \ \ (*) $$ where $X$ is adapted to the filtration ...
0
votes
0answers
11 views

How to do variable selection for Gradient boosting models like Xgboost and LightGBM

I am building a classification model with about ~110 variables and that gave me an AUC of about 71.96 on validation. I added about 10 more features and my AUC value decreased to 71.56 (which led to ...
0
votes
0answers
17 views

What is the relationship between stochastic mirror descent and stochastic gradient descent?

I don't know much about stochastic mirror descent and was wondering if someone could briefly summarize it in general terms and compare/contrast it to stochastic gradient descent. When I understand ...
0
votes
0answers
55 views

Non-linear optimization (curve fitting) with non-linear constraint

Similar to my question at Stackoverflow I want to calculate curve fitting coefficents of a non-linear function with some nasty inequality constraints. The results from my linked question are quite ...
0
votes
0answers
7 views

Implementation of Proximal alternating linearized minimization

The updates of the gradients are somehow wrong. I have implemented the below given algorithm. I have done something wrong ...
0
votes
0answers
5 views

Dynamic pricing with revenue optimization

I have been working on a case study where I have been given some data about fruits. Features include fruit_id, fruit_type, color, demand_supply_ratio, date_posted, geo_country, is_purchased, ...
0
votes
0answers
16 views

How can I determine what values of alpha and kappa to use for Bayesian Optimization?

I'm using the pretty great Bayesian Optimization package for python. I have a very noisy function I'd like to optimize for a given hyperparameter. I've read a little on this, and it seems like if ...
0
votes
1answer
30 views

Bootstrap and numerical optimization of statistic

Often times the bootstrap is used with a statistic that can be analytically evaluated (both in the real and the resampled datasets), e.g. the mean. But if the statistic can not be analytically ...
0
votes
0answers
32 views

Order of subtraction in gradient of squared-error

Consider a regression problem with the following loss function: $$ L(\theta) = (\hat{y}(\theta) - y)^2 $$ when doing gradient descent, we do $$ \theta \gets \theta - \alpha \nabla_\theta L(\theta) $$...
0
votes
0answers
10 views

How can we cast an optimisation problem as an inference problem?

The main idea of variational methods is to cast inference as an optimisation problem. In the paper Junction Tree Variational Autoencoder for Molecular Graph Generation, the authors state that the ...
0
votes
0answers
8 views

SMO alternative for support vector machine

I've used sequential minimal optimization (SMO) for solving the SVM dual formulation. I'm wondering if anyone has a good suggestion for an alternative algorithm for the same problem but with a ...
1
vote
1answer
28 views

What is the computational cost of gradient descent vs linear regression?

I know the computational costs for the closed form of linear regression is $O(n^3)$, but I can't find a similar cost comparison to gradient descent. There are some similar questions here with people "...
0
votes
1answer
15 views

Why is Hard-margin SVM training a minimization problem rather that a maximization problem?

I am looking at the wikipedia article for hard-margin SVMs and it looks like the optimization problem they use is "minimize ||w|| such that the classes are linearly separable" However isn't the ...
0
votes
1answer
16 views

Bayesian optimization integer set constraint

Given an ordered input set of boolean values $S$ of length $|S|$, I want to minimize a function $f(S)$ while maximizing the number of $True \in S$ constrained by $|True \in S|$ $\ge$ $threshold$ ...
0
votes
0answers
24 views

Why are analytical solutions possible for some optimization problems but not possible for others?

For example, an analytical solution is possible for linear regression (i.e. the normal equations) but it is not possible for logistic regression but logistic regression can be optimized with gradient ...
0
votes
0answers
14 views

Estimating weights in the assignment problem

How would you learn a function with the emphasis on feature interactions? I have the standard assignment problem: $$ \max_{x_{ij}} \sum_{(i, j)} w_{ij} x_{ij}, $$ where $w_{ij}$ is the weight of ...
0
votes
1answer
90 views

Combining Random forest with Adam (or an other gradient method)

There is no "gradient" in the standard Random Forest formulation, but can I combine random Forests with an optimisation method like Gradient Descent or SGD? Can I use Adam (Adaptive moment estimation)...
2
votes
0answers
25 views

Understanding Approximate Dynamic Programming

I am trying to write a paper for my optimization class about Approximate Dynamic Programming. I found a few good papers but they all seem to dive straight into the material without talking about the ...
0
votes
0answers
18 views

Optimal Power Transformation to Reduce Heteroscedasticity in Time Series

I would like to ask: what method should we employ if the variance in time series behaves like a high order (such as $au_t^5+bu_t^4+cu_t^3$) polynomial function with respect to mean? On internet, I ...
0
votes
0answers
19 views

ACER: optimization using the KKT conditions

In Page 5 Sample Efficient Actor-Critic with Experience Replay, the authors define an optimization problem with a linearized KL divergence constraint (Eq.11)as follow $$ \min_z{1\over 2}\Vert \hat g_t^...
0
votes
0answers
29 views

How to choose some discrete probability distribution to fit a target discrete probability distribution? [closed]

I have a set of vectors, each of which represents a discrete probability distribution, now I want to use some of them to simulate a certain discrete probability distribution. For instance, the ...
0
votes
1answer
25 views

Question about the location of regularization constant C in SVM

I've encountered very similiar but different functions in SVM optimization problem, the diffrence is in the location of regularization constant C. $\sum_{i=1}^n(1-(y_i(w^tx))_+ +\frac{1}{2C} \left\...
0
votes
2answers
23 views

Mysteriously defined KL-divergence term [duplicate]

I am trying to re-create a variational autoencoder. The loss function has two terms: reconstruction loss and KL-divergence term. KL-divergence is defined as $$ D_{KL}(P||Q) = -\sum_{x\in X}{P(X)\log\...
1
vote
0answers
17 views

Optimizing a piecewise-linear, convex function [closed]

I want to find the matrix $U$ that minimizes the following loss function: $$ \min_U( \max_{0 \le j \le t} (\operatorname{tr}(W_jU) + b_j) + \operatorname{tr}(U^TU)) $$ The given loss function is ...
0
votes
0answers
24 views

Minimizing expected loss with non-fixed probability distribution

Is there any convergence studies or algorithm to solve the following problem? $$ \mathbf{\hat{w}} = \min_\mathbf{w} \int\mathcal{L}(\mathbf{x};\mathbf{w})P(\mathbf{x};\mathbf{w})\ \mathrm{d}\mathbf{x}...
2
votes
0answers
14 views

Name for aggregate and component cost functions?

I was really thrilled to find a clear answer to Word for loss function except weight regularization? But I'm in a situation where we have multiple costs functions in the objective function. We ...
1
vote
0answers
37 views

Population Monte Carlo Algorithm using L2 Distance Measure/ Likelihood Distribution

I am currently struggling with some concepts of the Population Monte Carlo Framework. Initially, I came across this set of algorithms as I am currently trying to infer parameters from a 7D ...
1
vote
0answers
27 views

Gradient descent versus fixed point iteration

Fixed-point iteration Say I have the iteration $$x^{(k+1)} \leftarrow x^{(k)} + \alpha f(x^{(k)})$$ to find $x^\ast$, the root of $f$, i.e. $f(x^\ast)=0$, where $f:(a,b) \to \mathbb{R}$, $\exists ...
12
votes
5answers
2k views

Philosophical question on logistic regression: why isn't the optimal threshold value trained?

Usually in logistic regression, we fit a model and get some predictions on the training set. We then cross-validate on those training predictions (something like here) and decide the optimal threshold ...
0
votes
1answer
8 views

How to choose the best number of installments to sell an item

I have a dataset about previous sales. It contains both sold and cancelled item information. Which includes prices and number of installements for the item. Here I want to increase the probability of ...
1
vote
0answers
26 views

Momentum updates average of g, Adagrad also of g^2 - any other interesting updated averages for SGD convergence?

Updating exponential moving average is a basic tool of SGD methods, starting with of gradient $g$ in momentum method to extract local linear trend from the statistics. Then e.g. Adagrad, ADAM family ...
0
votes
0answers
15 views

Is my objective function to optimize svm parameters right? [migrated]

I have this objective function to optimize SVM parameters using PSO. ...
1
vote
0answers
26 views

Saddle-free Newton method for SGD - while Newton attracts saddles, is it worth to actively replel them?

While 2nd order methods have many advantages, e.g. natural gradient (e.g. in L-BFGS) attracts to close zero gradient point, which is usually saddle. Other try to pretend that our very non-convex ...
2
votes
1answer
40 views

On a mistake computing the Kullback Liebler Information Criterion

THE FRAMEWORK: Let $X_1$ be an observation from a normal random variable with mean zero and variance $\sigma^2$ and lets call the PDF $f(x)$. I want to minimize the Kullback Liebler Information ...
0
votes
0answers
8 views

Optimzation of Mutual Information among datasets

Consider $3$ datasets : $d_1$, $d_2$ and $d_3$. We calculate mutual information $M_1$ between $d_1$ and $d_3$ and $M_2$ between $d_2$, $d_3$. Given that the data in $d_3$ can be changed so that both ...
0
votes
1answer
35 views

Iterative optimization of alternative glm family

I'm setting up an alternative response function to the commonly used exponential function in poisson glms, which is called softplus and defined as $\frac{1}{c} \log(1+\exp(c \eta))$, where $\eta$ ...
2
votes
1answer
12 views

Model to Recommend Ideal Parameter Changes for Best Performance of Industrial Machine

I'm trying to develop a machine learning model to solve this problem, and am unsure of where to start. We begin with some user-defined settings. The settings are used by a machine to create a product....
1
vote
1answer
33 views

Are there are alternatives to gradient update rule?

Most optimization techniques (that I'm aware of) for non-linear cost functions that are commonly implemented rely on linearly updating a variable iteratively until a minimum is reached or a condition ...
4
votes
1answer
22 views

Continuous loss function that can measure one-side error

I am predicting a target $y$ using regression. In my application, the prediction $\hat{y}$ should be always no less than $y$. If $y>\hat{y}$, it is definitely a wrong prediction. On the $y<\hat{...
0
votes
1answer
18 views

Optimizing multiple objectives with different scales

I have multiple objectives, such as $f(\mathbf{x})$, $g(\mathbf{x})$, and $h(\mathbf{x})$. I would like to find a set of $x$ that can $\underset{\mathbf{x}}{argmin} [ f(\mathbf{x}) + g(\mathbf{x}) + ...
0
votes
0answers
14 views

Feature selection in parameter optimization

Let't say that $\theta$ is a vector or real numbers of the form $(\theta_1, \theta_2,\theta_3, ...,\theta_n)$ and $Obj(\theta)$ is a continuous function (objective function). Let's further say that I ...
0
votes
1answer
31 views

Constraints, bounds, and initialization variables in the GARCH / ARMA-GARCH models

I am interested in the correct way to estimate a GARCH/ARMA-GARCH model. I will refer to the coefficients as: ...
1
vote
0answers
24 views

Fitting Gaussian process with varying sample density

I have some underlying function of parameters $\theta_i$ that I'm trying to minimize. I sample this function using a latin hypercube and then, using some acquisition function, I obtain successive ...
0
votes
0answers
88 views

R - Moving average; MA(2), Maximum-Likelihood estimation through optim routine

I am trying to complete my assignment for time-series where I have to use Nile data to fit MA(2) model and estimate theta coefficients through creation of new function and optimizing it to get ...