Questions tagged [optimization]
Use this tag for any use of optimization within statistics.
0
votes
0answers
4 views
Stochastic gradient variance reduced methods
I'm doing stochastic gradient descent on a non-convex optimization problem. Gradient corresponds to an intractable expectation which I approximate via Monte Carlo averaging. I'm trying to infer the ...
0
votes
0answers
5 views
Fitting Gamma Distribution in Python with scalar factor
I have done optimization to minimize error of resulted model, however, the R-squared is still low. I try to multiply a scalar to the resulted model to obtain a better fit. The scalar will be generated ...
0
votes
0answers
9 views
Covariance of an estimate from optimization
Consider a standard linear regression model, $\boldsymbol y = X \boldsymbol \beta + \boldsymbol \epsilon$. $\boldsymbol y$ is a vector of $m$ responses, $X$ is a design matrix with $m$ rows and $p$ ...
0
votes
0answers
11 views
Linear Programming/optimisation with R? [on hold]
I hope this isn't a repeated question, but: does anybody have a firm idea of how much data can be handled by R when doing a linear optimisation model?
For example, will R be able to handle ...
0
votes
1answer
28 views
Singular gradient erros, NLS in R
I'm trying to fit nls(Mound~ a*kg.bag.collar^b + c, start = list(a = 83, b = -.5, c=100), data=test) using the dataset here. I've fit it without trouble without the ...
0
votes
0answers
19 views
How can I concentrate out parameters entering linearly in a partially non-linear regression?
I have model
$$y_i = x_i^\top\beta + \delta \exp(w_i\eta) + \epsilon_i$$
in setting up a non-linear regression problem
$$\min_{\beta,\delta,\eta} \frac{1}{2N} \sum_i^N (y_i - x_i^\top\beta - \delta \...
0
votes
0answers
24 views
An optimal stochastic problem and Monte-carlo
Assume we are in a Brownian filtration where I denote $W$ the Brownian motion. My problem is to numerically compute
$$
\min_X E (\int^1_0 X^2_tdt),\ \ \ \ (*)
$$
where $X$ is adapted to the filtration ...
0
votes
0answers
11 views
How to do variable selection for Gradient boosting models like Xgboost and LightGBM
I am building a classification model with about ~110 variables and that gave me an AUC of about 71.96 on validation. I added about 10 more features and my AUC value decreased to 71.56 (which led to ...
0
votes
0answers
17 views
What is the relationship between stochastic mirror descent and stochastic gradient descent?
I don't know much about stochastic mirror descent and was wondering if someone could briefly summarize it in general terms and compare/contrast it to stochastic gradient descent.
When I understand ...
0
votes
0answers
55 views
Non-linear optimization (curve fitting) with non-linear constraint
Similar to my question at Stackoverflow I want to calculate curve fitting coefficents of a non-linear function with some nasty inequality constraints. The results from my linked question are quite ...
0
votes
0answers
7 views
Implementation of Proximal alternating linearized minimization
The updates of the gradients are somehow wrong.
I have implemented the below given algorithm. I have done something wrong
...
0
votes
0answers
5 views
Dynamic pricing with revenue optimization
I have been working on a case study where I have been given some data about fruits. Features include fruit_id, fruit_type, color, demand_supply_ratio, date_posted, geo_country, is_purchased, ...
0
votes
0answers
16 views
How can I determine what values of alpha and kappa to use for Bayesian Optimization?
I'm using the pretty great Bayesian Optimization package for python. I have a very noisy function I'd like to optimize for a given hyperparameter.
I've read a little on this, and it seems like if ...
0
votes
1answer
30 views
Bootstrap and numerical optimization of statistic
Often times the bootstrap is used with a statistic that can be analytically evaluated (both in the real and the resampled datasets), e.g. the mean.
But if the statistic can not be analytically ...
0
votes
0answers
32 views
Order of subtraction in gradient of squared-error
Consider a regression problem with the following loss function:
$$
L(\theta) = (\hat{y}(\theta) - y)^2
$$
when doing gradient descent, we do
$$
\theta \gets \theta - \alpha \nabla_\theta L(\theta)
$$...
0
votes
0answers
10 views
How can we cast an optimisation problem as an inference problem?
The main idea of variational methods is to cast inference as an optimisation problem.
In the paper Junction Tree Variational Autoencoder for Molecular Graph Generation, the authors state that the ...
0
votes
0answers
8 views
SMO alternative for support vector machine
I've used sequential minimal optimization (SMO) for solving the SVM dual formulation. I'm wondering if anyone has a good suggestion for an alternative algorithm for the same problem but with a ...
1
vote
1answer
28 views
What is the computational cost of gradient descent vs linear regression?
I know the computational costs for the closed form of linear regression is $O(n^3)$, but I can't find a similar cost comparison to gradient descent.
There are some similar questions here with people "...
0
votes
1answer
15 views
Why is Hard-margin SVM training a minimization problem rather that a maximization problem?
I am looking at the wikipedia article for hard-margin SVMs and it looks like the optimization problem they use is
"minimize ||w|| such that the classes are linearly separable"
However isn't the ...
0
votes
1answer
16 views
Bayesian optimization integer set constraint
Given an ordered input set of boolean values $S$ of length $|S|$,
I want to minimize a function $f(S)$
while maximizing the number of $True \in S$
constrained by $|True \in S|$ $\ge$ $threshold$
...
0
votes
0answers
24 views
Why are analytical solutions possible for some optimization problems but not possible for others?
For example, an analytical solution is possible for linear regression (i.e. the normal equations) but it is not possible for logistic regression but logistic regression can be optimized with gradient ...
0
votes
0answers
14 views
Estimating weights in the assignment problem
How would you learn a function with the emphasis on feature interactions?
I have the standard assignment problem:
$$
\max_{x_{ij}} \sum_{(i, j)} w_{ij} x_{ij},
$$
where $w_{ij}$ is the weight of ...
0
votes
1answer
90 views
Combining Random forest with Adam (or an other gradient method)
There is no "gradient" in the standard Random Forest formulation, but can I combine random Forests with an optimisation method like Gradient Descent or SGD?
Can I use Adam (Adaptive moment estimation)...
2
votes
0answers
25 views
Understanding Approximate Dynamic Programming
I am trying to write a paper for my optimization class about Approximate Dynamic Programming. I found a few good papers but they all seem to dive straight into the material without talking about the ...
0
votes
0answers
18 views
Optimal Power Transformation to Reduce Heteroscedasticity in Time Series
I would like to ask: what method should we employ if the variance in time series behaves like a high order (such as $au_t^5+bu_t^4+cu_t^3$) polynomial function with respect to mean? On internet, I ...
0
votes
0answers
19 views
ACER: optimization using the KKT conditions
In Page 5 Sample Efficient Actor-Critic with Experience Replay, the authors define an optimization problem with a linearized KL divergence constraint (Eq.11)as follow
$$
\min_z{1\over 2}\Vert \hat g_t^...
0
votes
0answers
29 views
How to choose some discrete probability distribution to fit a target discrete probability distribution? [closed]
I have a set of vectors, each of which represents a discrete probability distribution, now I want to use some of them to simulate a certain discrete probability distribution. For instance, the ...
0
votes
1answer
25 views
Question about the location of regularization constant C in SVM
I've encountered very similiar but different functions in SVM optimization problem, the diffrence is in the location of regularization constant C.
$\sum_{i=1}^n(1-(y_i(w^tx))_+ +\frac{1}{2C} \left\...
0
votes
2answers
23 views
Mysteriously defined KL-divergence term [duplicate]
I am trying to re-create a variational autoencoder. The loss function has two terms: reconstruction loss and KL-divergence term. KL-divergence is defined as $$ D_{KL}(P||Q) = -\sum_{x\in X}{P(X)\log\...
1
vote
0answers
17 views
Optimizing a piecewise-linear, convex function [closed]
I want to find the matrix $U$ that minimizes the following loss function:
$$ \min_U( \max_{0 \le j \le t} (\operatorname{tr}(W_jU) + b_j) + \operatorname{tr}(U^TU)) $$
The given loss function is ...
0
votes
0answers
24 views
Minimizing expected loss with non-fixed probability distribution
Is there any convergence studies or algorithm to solve the following problem?
$$
\mathbf{\hat{w}} = \min_\mathbf{w} \int\mathcal{L}(\mathbf{x};\mathbf{w})P(\mathbf{x};\mathbf{w})\ \mathrm{d}\mathbf{x}...
2
votes
0answers
14 views
Name for aggregate and component cost functions?
I was really thrilled to find a clear answer to Word for loss function except weight regularization? But I'm in a situation where we have multiple costs functions in the objective function. We ...
1
vote
0answers
37 views
Population Monte Carlo Algorithm using L2 Distance Measure/ Likelihood Distribution
I am currently struggling with some concepts of the Population Monte Carlo Framework. Initially, I came across this set of algorithms as I am currently trying to infer parameters from a 7D ...
1
vote
0answers
27 views
Gradient descent versus fixed point iteration
Fixed-point iteration
Say I have the iteration
$$x^{(k+1)} \leftarrow x^{(k)} + \alpha f(x^{(k)})$$
to find $x^\ast$, the root of $f$, i.e. $f(x^\ast)=0$, where $f:(a,b) \to \mathbb{R}$, $\exists ...
12
votes
5answers
2k views
Philosophical question on logistic regression: why isn't the optimal threshold value trained?
Usually in logistic regression, we fit a model and get some predictions on the training set. We then cross-validate on those training predictions (something like here) and decide the optimal threshold ...
0
votes
1answer
8 views
How to choose the best number of installments to sell an item
I have a dataset about previous sales. It contains both sold and cancelled item information. Which includes prices and number of installements for the item. Here I want to increase the probability of ...
1
vote
0answers
26 views
Momentum updates average of g, Adagrad also of g^2 - any other interesting updated averages for SGD convergence?
Updating exponential moving average is a basic tool of SGD methods, starting with of gradient $g$ in momentum method to extract local linear trend from the statistics.
Then e.g. Adagrad, ADAM family ...
0
votes
0answers
15 views
Is my objective function to optimize svm parameters right? [migrated]
I have this objective function to optimize SVM parameters using PSO.
...
1
vote
0answers
26 views
Saddle-free Newton method for SGD - while Newton attracts saddles, is it worth to actively replel them?
While 2nd order methods have many advantages, e.g. natural gradient (e.g. in L-BFGS) attracts to close zero gradient point, which is usually saddle. Other try to pretend that our very non-convex ...
2
votes
1answer
40 views
On a mistake computing the Kullback Liebler Information Criterion
THE FRAMEWORK:
Let $X_1$ be an observation from a normal random variable with mean zero and variance $\sigma^2$ and lets call the PDF $f(x)$.
I want to minimize the Kullback Liebler Information ...
0
votes
0answers
8 views
Optimzation of Mutual Information among datasets
Consider $3$ datasets : $d_1$, $d_2$ and $d_3$. We calculate mutual information $M_1$ between $d_1$ and $d_3$ and $M_2$ between $d_2$, $d_3$.
Given that the data in $d_3$ can be changed so that both ...
0
votes
1answer
35 views
Iterative optimization of alternative glm family
I'm setting up an alternative response function to the commonly used exponential function in poisson glms, which is called softplus and defined as $\frac{1}{c} \log(1+\exp(c \eta))$, where $\eta$ ...
2
votes
1answer
12 views
Model to Recommend Ideal Parameter Changes for Best Performance of Industrial Machine
I'm trying to develop a machine learning model to solve this problem, and am unsure of where to start.
We begin with some user-defined settings. The settings are used by a machine to create a product....
1
vote
1answer
33 views
Are there are alternatives to gradient update rule?
Most optimization techniques (that I'm aware of) for non-linear cost functions that are commonly implemented rely on linearly updating a variable iteratively until a minimum is reached or a condition ...
4
votes
1answer
22 views
Continuous loss function that can measure one-side error
I am predicting a target $y$ using regression. In my application, the prediction $\hat{y}$ should be always no less than $y$. If $y>\hat{y}$, it is definitely a wrong prediction. On the $y<\hat{...
0
votes
1answer
18 views
Optimizing multiple objectives with different scales
I have multiple objectives, such as $f(\mathbf{x})$, $g(\mathbf{x})$, and $h(\mathbf{x})$.
I would like to find a set of $x$ that can $\underset{\mathbf{x}}{argmin} [ f(\mathbf{x}) + g(\mathbf{x}) + ...
0
votes
0answers
14 views
Feature selection in parameter optimization
Let't say that $\theta$ is a vector or real numbers of the form $(\theta_1, \theta_2,\theta_3, ...,\theta_n)$ and $Obj(\theta)$ is a continuous function (objective function). Let's further say that I ...
0
votes
1answer
31 views
Constraints, bounds, and initialization variables in the GARCH / ARMA-GARCH models
I am interested in the correct way to estimate a GARCH/ARMA-GARCH model.
I will refer to the coefficients as:
...
1
vote
0answers
24 views
Fitting Gaussian process with varying sample density
I have some underlying function of parameters $\theta_i$ that I'm trying to minimize. I sample this function using a latin hypercube and then, using some acquisition function, I obtain successive ...
0
votes
0answers
88 views
R - Moving average; MA(2), Maximum-Likelihood estimation through optim routine
I am trying to complete my assignment for time-series where I have to use Nile data to fit MA(2) model and estimate theta coefficients through creation of new function and optimizing it to get ...
