Questions tagged [optimization]
Use this tag for any use of optimization within statistics.
0
votes
0answers
19 views
Writing equation of Neural Network model
I am very new to neural networks. My goal is to replace my multivariate regression model with NN and use its output to perform mathematical optimization in order to identify the values of the input ...
0
votes
1answer
74 views
What is the Maximum Likelihood Estimator of $f(x;\theta) = \frac{3\theta x^{3\theta -1}}{(1+x^{3})^{\theta +1}} $
$f(x;\theta) = \frac{3\theta x^{3\theta -1}}{(1+x^{3})^{\theta +1}}, x>0, \theta>0 $
I came up with FOC:
$ \hat{\theta} = \frac{1}{-3\log(x)+\log(1+x^3)} $
Is this correct? Thanks:-)
I took ...
0
votes
0answers
9 views
How to get precise answer from stochastic gradient descent
I have a convex optimization problem in few variables and I have an unbiased estimator of the gradient without having the ability to evaluate the function itself. I want to do gradient descent but the ...
1
vote
0answers
9 views
Should one track the loss or accuracy of a neural network when training it?
Should one track a model's progress using its loss or its accuracy? I ask this because sometimes the loss at a epoch is higher than that at previous epochs (which is a bad thing) but so is the ...
1
vote
0answers
22 views
Can Expectation-Maximization algorithm estimate parameters other than mean and variance (from a model distribution)?
We know that we can use Expectation-Maximization algorithm to estimate parameters from a Gaussian mixture model, say $\mu$, $\sigma$, and $\phi$ (they are parameters of the Gaussian distributions)as ...
0
votes
0answers
22 views
Are there any optimizers that perform well on small datasets?
With regards to neural networks (or any optimization-based model in general), are there any optimizers that excel in the small dataset regime? Most optimizers I know of require large amounts of data ...
0
votes
0answers
5 views
A simple method for finding the $n$ most distinctive shapes in $x \times y$ pixels region?
Suppose that we have an area made of $x\times y$ pixels. By turning those $x\times y$ many pixels on and off, we can create $2^{x\times y}$ many possible shapes.
But obviously many of those $2^{x\...
0
votes
0answers
5 views
How does Aspiration Criteria not lead to looping?
So in Tabu Search, we keep a tabu list of recently visited sites and restrict access. We also have "Aspiration Criteria" which allows us to override tabu list and move to that new point anyways.
My ...
0
votes
0answers
7 views
Initial simplex for the Nelder-Mead algorithm in R using the “optim” function
How does R create the initial simplex when using the "optim" function? Is it different to the method applied in Matlab's "fminsearch"? Are there any other differences in the implementation?
I have ...
0
votes
0answers
13 views
Wassertein “least squares” and symmetries
So here's a scenario: I have points $(\mu_1^j,\mu_2^j)$ and I associated them the following distribution $$\rho_j=1/2\delta_{\mu_1^j}+1/2\delta_{\mu_2^j}$$
These have symmetry (exchanging $\mu_1^j$ ...
0
votes
0answers
15 views
Finding optimal parameters for an unknown function - (learning with an unknown loss function)
I am working on a project where my objective is to find a set of parameters for a synthesizer which makes the synthesizer replicate an input sound. I do not know exactly what goes on inside of the ...
5
votes
2answers
490 views
How well should I expect Adam to work?
I've been coding up a neural network package for my own amusement, and it seems to work. I've been reading about Adam and from what I've seen it's very difficult to beat.
Well, when I implement the ...
0
votes
1answer
26 views
Optimisation by using directional derivative
So I’ve seen the code of an R package where a two dimensional optimisation (actually MLE, finding the minimum of the negative log likelihood) is performed with the optim function and also two optimise ...
0
votes
0answers
24 views
Parameters estimation using Kalman filter
I have a model like this:
$$P_t=\alpha_t+gP_{t-1}+u_t$$
$$\alpha_t=\alpha_{t-1}+d+n_t$$
Where {$u_t$} and {$n_t$} are normal, iid with 0 mean but unknown variance.
I want to estimate the parameters $...
0
votes
3answers
47 views
Neural Network for input values optimization
I have electric machine, which parameters I measure by 10 sensors. 8 of them measures "input" values and 2 of them result (output). I've got tons of historical data of all of these sensors. I built a ...
0
votes
0answers
22 views
Optimizer for $L_1$ (LASSO) penalty: L-BFGS, COBYLA, OWL-QN or other?
I am sure there are other sources, but reading the following vignette of R package lbfgs, it is claimed that L-BFGS optimization algorithm is not suited for an ...
0
votes
1answer
40 views
Can most optimization problems be framed and tackled as reinforcement learning problems?
There is a clear overlap between both. Which characteristics can help us identify problems that could be tackled as classic optimization and rf also?
0
votes
0answers
12 views
Gradient Descent method for Wolfe Duality Hyperplane Optimization
I am troubling myself by learning how to build an optimal hyperplane for a separable case using the $\texttt{iris}$ data in R.
The function I am trying to maximize w.r.t $\alpha_{i}$ is:
$$
L_D = \...
1
vote
0answers
35 views
How positive definite Hessian approximations for SGD (e.g. Gauss-Newton) handle saddles?
For example due to symmetry of parameters, functions optimized in machine learning usually have huge number of local minima and saddles - growing exponentially with dimension.
I am trying to ...
0
votes
0answers
25 views
Rank 1 SVD with constraint on U
I need to perform a particular rank 1 decomposition of a sparse matrix $\mathbf{A} \in \mathbb{R}^{n\times n}$.
In particular I am looking for the positive vector $\mathbf{u} \in \mathbb{R}^{+n}$ ...
0
votes
2answers
56 views
Variance/bias trade off regularisation penalty - why does it take this form?
In machine learning, if we estimate weights using a loss function
$$L(W) = ||Y-F_W(X)||^2$$
(where $W$ is a weight matrix) we may add a "regularisation penalty" to control for the "variance/bias ...
2
votes
0answers
43 views
What to treat as (hyper-)parameter and why
I have been wondering about the differences between model paramters and model hyperparameters, as well as what their categorization means for a learning problem.
Is the distinction between model ...
1
vote
0answers
35 views
Which optimization algorithm applies better for this production problem? Cost minimization
This is a cost minimization problem, where I have to plan the development of a field and the installation of some machines there.
Rectangles A,B,C,D represent production areas, that can overlap. I ...
2
votes
0answers
18 views
How to solve a non-convex with equality constraint optimization problem?
I have a non-convex optimization problem with equality constraint, I can derive the KKT conditions, but it seems just one of the KKT conditions is valid. Could you please give some advice on how to ...
0
votes
1answer
25 views
Can one reverse-engineer and deduce the underlying data for a constrained (max/min) optimization problem?
If one has the result, the constraints are known, but one does not have the input data.
0
votes
0answers
15 views
Are there any generally simple examples on Tabu Search in R?
I am looking for any examples of implementing Tabu Search in R.
I know there is a package, but I would like to see if there are any good instructive examples where the code is built up and used to ...
4
votes
1answer
49 views
Do there exist adaptive step size methods for Newton-Raphson optimization?
Stochastic/Mini-batch gradient descent, caused by interest in deep learning, has made lots of advances in adaptive step sizes. For example, Adam, Nadam, Adamax, ..., are all improvements to the ...
1
vote
1answer
42 views
Select optimal points for Gaussian process with a well-known target function
I'm currently trying to select the optimal points for a Gaussian Process Regression, and the important thing is that i already know the whole target function.
Therefore, it's not Online Learning ...
0
votes
0answers
8 views
'Shrink' Step occuring often in Nelder Mead optimization?
I have an implementation of Nelder Mead, which is giving me good results. I had a bug, though, and while I managed to fix that bug, I noticed during my debugging that the 'shrink' step is occurring ...
0
votes
0answers
7 views
how to set up an optimization problem to split a group of people into two groups, with several constraints
I am a bit stuck with this problem; I found a temporary (an perhaps suboptimal) solution using Excel, but I'd like to hear your opinion /advice, please.
9 people want to form a group and go on on a ...
0
votes
0answers
15 views
Minimizing numerically a nondifferentiable function
Many (likelihood) functions are not differentiable at the optimal point. Does this cause problems
a) in the numerical methods used to minimize the function based on the gradient?
b) in the statistical ...
1
vote
0answers
25 views
Optimize an objective based on a trained model
I want to find a joint optimal subset based on individual scores from a predictive model.
Example:
Say I have a set of customers and a set of products. And I have trained a model for predicting the ...
0
votes
0answers
5 views
Recommendations for Data Fitting Mean Reversion Processes
I have a time-series with clear mean-reverting properties over some time-scale. I have a very long measurement of this series, so can see that, whilst it always reverts to a fixed mean, the ...
0
votes
1answer
34 views
Determining error between two surfaces given same discrete inputs?
Apologies if this isnt the best SE forum to ask on, but it seems relevant here.
I have, as an output of a machine learning algorithm, a surface in z, which has known increments along x and y. These ...
0
votes
0answers
37 views
R - matrix optimization problem (Lee-Carter with MLE parameters)
I am trying to reproduce one of models evolved on the base of Lee-Carter model by use of RStudio. Because I am still pretty fresh in this software, my question is not really sophisticated.
Applying ...
0
votes
0answers
13 views
How to choose between dual gradient descent and the method of Lagrangian multipliers?
For an optimization problem
$$
\max f(x)\\\
s.t. g(x)\le 0
$$
The Lagrangian is
$$
\mathcal L(x, \lambda)=f(x)-\lambda g(x)
$$
Dual gradient descent solves it by (according to Page 43 of this lecture,...
1
vote
0answers
36 views
Is the Franke-Wolfe algorithm the same as Manifold optimization?
The Frank-Wolfe optimization algorithm describes optimization over a constrained domain.
In the Manifold Optimization literature (e.g. [1]) a Gradient-Step is done using an exponential map. This maps ...
2
votes
1answer
21 views
What is meant by steepest ascent
I am trying to understand the following from my notes, in relation to Newton Like methods.
if
$x^{(t+1)}=x^{(t)}-(M^{(t)})^{-1}g'(x^{(t)})$
is not guarnteed to be uphill.
However if $M^{(t)}=-I$ (...
0
votes
0answers
11 views
Sample correlation as the loss function for multiple linear regression [duplicate]
I have been thinking about the problem but I need some guidance. Consider a multiple linear regression problem.
Instead of minimizing the mean squared errors (MSE), I want to my loss function to be ...
3
votes
1answer
336 views
Why second order SGD convergence methods are unpopular for deep learning?
It seems that, especially for deep learning, there are dominating very simple methods for optimizing SGD convergence like ADAM - nice overview: http://ruder.io/optimizing-gradient-descent/
They trace ...
0
votes
1answer
17 views
How to predict similarity of unseen data to the training set?
I have a time series of human pose data which are recorded from real humans.
I want to train the model with unsupervised learning on the training data.
Let's call this the "real" training data.
The ...
2
votes
0answers
27 views
Analyse sensitivity of hyper-parameters of Machine Learning Models
I want to analyse how sensitive my non neural net machine learning models are to the choice of the different parameters. I am currently using grid search to tune the models. Is there any method that I ...
0
votes
0answers
38 views
Optimization of a drone given a cost function
Currently, I have a drone that has 6 motors. Each motor takes a different amount of power to run. The objective of the drone is to travel at a constant desired velocity, with the minimum amount of ...
1
vote
1answer
40 views
Convex optimization: Is gradient descent faster if a regularizer is added?
I am not sure if this is a true statement or not but there appears to be an intuition around this among experts in the field that I do not quite understand.
The idea is: Given a convex optimization ...
3
votes
1answer
35 views
Solve the optimization problem of tree, should we make each rectangle contains exactly one training data point?
I was reading the book "An Introduction to Statistical Learning with Applications in R". In page 306, when talking about the objective function of tree model, the book says:
"The goal is to find ...
1
vote
1answer
36 views
Relationship between first and second order condition of convexity
Suppose we have a function $f(\boldsymbol{x})$ and its hessian, i.e $\nabla_{\boldsymbol{x}}^2f(\boldsymbol{x})$, equals $\mathbf{0}$.
We know that for convexity $\nabla_{\boldsymbol{x}}^2f(\...
1
vote
0answers
52 views
Maximum likelihood estimator of the parameter of randomness in Watts and Strogatz's model (1998)
According to the paper
Menezes, M. B., Kim, S., & Huang, R. (2017). Constructing a Watts-Strogatz network from a small-world network with symmetric degree distribution. PloS one, 12(6), e0179120,...
2
votes
2answers
110 views
When is the optimizer of $\mathbb E[X]$ and $\mathbb E[X^2]$ the same?
Consider a non-negative random variable $X\sim p(\theta)$, that is, following distribution $p$ parametrized by $\theta$.
Suppose we find a value of the parameters $\theta^*$ such that $$\mathbb E_{X\...
2
votes
0answers
9 views
optimal scheduling allocation between online and walk-in queues
I have a scheduling queue that can be split between walk-ins and online booking appointments. In order to serve these queues I have a limited number of resources available. What is the optimal way to ...
3
votes
0answers
36 views
Most efficient LAD solver
What is the most efficient way to solve linear Least absolute deviation regression problem?
I know it can be solved using linear programming, is there a better/faster method?
Edit: I'm interested ...
