Questions tagged [optimization]
Use this tag for any use of optimization within statistics.
0
votes
1answer
28 views
Why multiply an error by itself minus one? [on hold]
I'm trying to understand how the optimisation is working in an algorithm. In support of the optimisation, errors are calculated.
For one type of error, before it returns the value, it multiplies it ...
1
vote
0answers
18 views
Strange rugarch error: $ operator invalid for atomic vectors? [on hold]
So, I am trying to make a huge nested for loop (optimizations be left for later) to fit all of the GARCH models available from rugarch.
This is my MLE that ...
0
votes
0answers
14 views
optimization algorithm [on hold]
optimization algorithm which having lot of constraints and the output should be binary either zero or one. The algorithm should satisfy linear and non linear constraints.Which one is better other than ...
2
votes
0answers
18 views
Validation ROC AUC not improving with validation cross-entropy loss?
I am training a neural network that is doing binary image classification on several thousand images. I am running 5 fold cross validation (train on 4, validate on 1) with cross entropy (CE) loss. I am ...
0
votes
0answers
29 views
Regularization of linear regression problem [duplicate]
Consider a vector $a \in R^n$. I want to know how I can find analytically the solution of the following optimization problem: $x^* = argmin_{x \in R^n} f(x)$, where $f(x) = ||x-a||_{2}^2 + \lambda ||x|...
0
votes
0answers
7 views
Optimal multivariate binning where the cut-points must be the same for all observations
I have a large data set with many discrete and continuous variables. All the variables are present in every observation. I want to explain (the log of) one continuous variable using all the other ...
0
votes
0answers
24 views
what does the notation $X^{(n)}$ mean in the context of matrices? [on hold]
What does the notation $x^{(n)}$ where x is a matrix and n is an integer?
0
votes
0answers
8 views
Sound unit testing of convergence of an implementation of a stochastic algorithm
(My statistics knowledge comes mainly from computer science, so I might be overthinking the following...)
I have implementations of several stochastic optimization algorithms (evolutionary algorithms,...
0
votes
0answers
10 views
High error value when estimating model parameters
I have a non linear system of ODEs and to estimate 4 of the model parameters I am using Matlab fmincon by minimising the sum of squared errors (SSE). I have only 5 ...
0
votes
0answers
27 views
Using Random Forest for Optimisation
Can random forests be used for optimisation?
I created a random forest which predicts prices, customers will pay. Lets assume I want to maximize the price a customer will pay. To do this I have 3 ...
1
vote
0answers
22 views
Could machine learning be used to select best parameter for more than one loop optimization?
I do not know if i can ask this question here or not. But I really need it.
I'm very new to ML/DL/NN field.
I have seen many articles tackling the problem of selection of the best parameter for loop ...
0
votes
0answers
7 views
how to separately optimize the parameters of convolutional layers and the parameters of fully connected layers [closed]
I want to use two optimizers to optimize the parameters of the CNN (convolutional layers and fully connected layers) in ...
0
votes
0answers
18 views
Infimum of the difference between KL Divergence and Entropy [closed]
Suppose $y_1 = 1, y_2 = y_3 = ...=y_n = 0\ (n\geq2)$, and $\sum_{i=1}^np(y_i|x;\theta) = 1$, $0\leq p(y_i|x;\theta)\leq1$. Meanwhile $\alpha > 0$ is a constant.
Let's define a function
$$f(\theta)...
0
votes
0answers
22 views
Coordinate descent in integer programing: when does it work?
Denote $N_i=\{0,1,\dots,\bar{n}_i\}$ and define $N=N_1\times \dots \times N_I$. I want to minimize a function $f:N\rightarrow \mathbb{R}$. For the functions $f$ that interest me, it is very easy to ...
0
votes
0answers
9 views
Difference between Surrogate Modelling and Surrogate Optimization
Based on my understanding of Surrogate Optimization, the goal is to find the best set of parameters of a low fidelity model, by continuously increasing sampling in the domain most likely to have the ...
0
votes
0answers
27 views
why Newton's method is not sensitive to ill-conditioned Hessian?
On Jorge Nocedal's , Page 501, "This property alone is enough to make many unconstrained minimization algorithms such as quasi-Newton and conjugate gradient perform poorly. Newton’s method, on the ...
0
votes
0answers
7 views
How to differentiate through the optimization process in Model-Agnostic Meta-Learning?
I'm reading the paper introducing Model-Agnostic Meta-Learning (MAML).
As far as I understood, there is a nested optimization process in MAML. Given a task $\tau_i$ sampled from distribution $p(\tau)...
0
votes
0answers
24 views
Autoencoder as an optimization (search) problem
We all know that machine learning problems can be modeled as an optimization problem where we are searching for the best set of parameter values in the parameter space that optimizes our objective ...
3
votes
0answers
34 views
regression analysis with both slope and value data
I have measured data $\{x_i, y_i, y'_i\}$, to which I would like to fit a polynomial $y=a x^2 + bx + c$ and $y' = 2ax + b$.
It occurs to me that the regression problem of fitting $y=a x^2 + bx +c$ to ...
2
votes
1answer
43 views
Newsvendor problem with Poisson demand
Problem:
The demand for a particular weekly magazine in a newsstand follows a Poisson distribution averaging 5.2 copies per week. The value paid for each magazine is 15.00 and the sale price is 30.00....
0
votes
0answers
21 views
Optimization Technique Needed
I am trying to figure out an optimization technique to below:
For context, everything to the left of "New Upcoming Games" is historical data. In my actual dataset I have about 200-300 rows that will ...
0
votes
0answers
10 views
How to choose the best parameter for the LINEX loss function?
I am using a LINEX loss function to evaluate my forecast. What procedure should I follow to find the best $\alpha$ parameter?
LINEX function: $$L(e) = \exp(\alpha e) - \alpha e - 1$$
Where $e$ is the ...
2
votes
1answer
34 views
Weight optimization in order to maximize correlation [R]
I'm researching a way to speed up a problem I already have in Excel's Solver.
I wanna know if my approach is correct (I doubt it tbh) and if there're better alternatives.
I have a matrix of scores I ...
0
votes
1answer
12 views
Does using random minibatchs give more resilance against local minima, vs full batch gradient descent
It was my believe that one of the advantages of using minibatches,
when training a neural network via gradient descent (be it "vanilla" or the latest flavour of AdaGrad),
was an increased resiliance ...
0
votes
0answers
21 views
Are there any good, general optimization algorithms for nonlinear regression in R?
I have generated data similar to a model I want to build:
...
3
votes
0answers
15 views
How sensitive is $L_p$ regression to initialisation?
Consider that I wish to solve a linear regression in the $L_p$ framework
That is, the optimisation problem that I wish to solve is
$$ \mathbf{w} = \text{argmin}_{\mathbf{w}} ||\mathbf{w}^T\mathbf{x} -...
0
votes
1answer
24 views
SVM optimization problem: question about constraint
I am studying SVM algorithm and its optimization problem.
When we are constructing optimization problem, we say, that we are searching for such separating hyperplane, so that we rescale $w$ and $b$, ...
0
votes
0answers
11 views
Method to objectively optimize several parameters to find best fit between 2 curves
I'm not sure if the title describes correctly what I'm asking, it's hard to put into few words what I need, and my knowledge in statistics is very basic.
I have an observed curve of, let's say, the ...
0
votes
1answer
55 views
How to perform batch training using L-BFGS?
I want to train a neural network for regression.
The neural network is actually composed of 4 separate child neural networks, each child neural network has a layer structure of
...
0
votes
0answers
18 views
Optimization technique to find the minimum change required in features to cause change of classifier prediction
Assuming a dataset has N features and 4 (P,Q,R,S) categories. A classifier is to predict which of the 4 categories a given datapoint belongs.
Say a classifier predicts that some datapoint belongs to ...
2
votes
0answers
25 views
Derivatives of quantile loss function [duplicate]
I'm reading a text - Roger Koenker (2005) Quantile Regression [page 8] - that goes like this:
Consider the function $$R(\xi) = \sum_{i=1}^n \rho_\tau(y_i-\xi)$$
where $$\rho_\tau(y_i-\xi) =(y_i-\xi)(\...
6
votes
2answers
103 views
Neural network's weight reduction
Are there any algorithms/methods for taking a trained model and reducing its number of weights with as little negative effect as possible to its final performance?
Say I have a very big (too big) ...
2
votes
0answers
14 views
How to identify manifolds for an optimisation problem
I don't have much experience in topology, but I am interested to know if:
• Given a particular problem and associated cost function, how would one deduce what kind of manifold this problem lies on.
...
0
votes
0answers
6 views
How gradient decent training will affect if we use feature-crosses or high-order polynomials?
Considering Multivariate linear regression. We use feature scaling + mean normalization(feature transformation) on our features to keep them on the same scale. If we don't do that then our contour ...
1
vote
0answers
44 views
How cost function for simple linear regression behaves under different settings with batch gradient descent? [closed]
In the linear regression problem, using a simple linear model with 1 variable & with 2 model parameters, performing batch Gradient Descent(GD) & assuming I am using Mean Square error as my ...
0
votes
1answer
16 views
How do I design this multi-objective fitness function for use with genetic algorithm?
I have a multi-objective optimization problem that I am applying genetic algorithm (GA) to solve. Currently, there are only 2 objectives:
minimize cost
maximize validity
The cost minimization is ...
1
vote
1answer
15 views
Word for loss function except weight regularization?
A typical loss function in machine learning is:
$$L(\theta,x) = \mathcal L(\theta,x) +\sum_{\theta} |\theta|$$
I typically use the word “loss function” both for $L(\theta,x)$ and for $\mathcal L(\...
0
votes
0answers
8 views
Dataset that overfits with increasing test error curve
I want to test some optimization algorithm for DNN to see if it prevents overfitting. I'm looking for a images dataset that overfits with SGD s.t. test 0-1 error curve increases with iterations. If it ...
0
votes
1answer
47 views
Total variation regularization in deep learning
For current deep learning models, we can find basically two kinds of regularization on:
Activation
Weights
The common $L_1$ and $L_2$ on weights can lead to a MAP problem where the regularization ...
2
votes
0answers
28 views
Justification of acceptance probability in simulated annealing
In simulated annealing the acceptance probability for a new state in step $k$ is traditionally defined as
$$ P(\text{accept new})= \begin{cases}
\exp(-\frac{\Delta}{T_k}), & \text{ if } \Delta \...
0
votes
0answers
17 views
How can I not show the initialization of the estimation in the Extended Kalman Filter?
I'm making estimates through the Extended Kalman Filter and I have a problem related to the vertical axis of my figure, it's too big, so I can not see population dynamics. However, I wish it did not ...
1
vote
0answers
32 views
How do we know we're maximizing the Lagrangian objective function in PCA?
In Principal Component Analysis, we start with $m$ observations $x_1,\dots,x_m$, each of which is an $n$-dimensional vector. Assume we have centered the data; that is, we have subtracted the variable ...
0
votes
1answer
26 views
Assignment Problem (Linear Programming, Genetic Algorithm, etc.)
I'm looking for advice on how I should approach a specific problem. Some background first:
The problem is about shipments falling into a bin. There are 19 such bins, which are further sorted into 20 ...
8
votes
1answer
312 views
Why can't a single ReLU learn a ReLU?
As a follow-up to My neural network can't even learn Euclidean distance I simplified even more and tried to train a single ReLU (with random weight) to a single ReLU. This is the simplest network ...
2
votes
0answers
60 views
Linear programming optimization [closed]
I have found through my reading that applying linear programming optimization techniques are substantially more expensive compared to mean squared error-based methods.
Could someone please help in ...
0
votes
1answer
119 views
How to use optim function in R on my custom Residual Sum of Squares function?
I'm trying to use the optim function in R to find the optimal values for $\alpha$ and $l_0$ for my custom residual sum of squares (RSS) function, but the values I'm ...
0
votes
0answers
37 views
How to select parameters for ADAM gradient descent
I am using ADAM for performing gradient descent. I am having difficulty in setting the learning rate, $\beta_1$ and $\beta_2$. Along with gradient descent, I am projecting the paramters on $L_1$ ball ...
0
votes
0answers
13 views
Methods for Large Optimal Assignment problems with sparse connections
I have an assignment problem involving about 2 million workers and 2 million jobs. Thus the cost matrix has 4x10^12 elements. A standard approach would be the Hungarian algorithm (Munkres Algorithm), ...
0
votes
0answers
45 views
Optimization options to minimize mean absolute error when model is a Neural network
Lately I've seen some advantages mostly in model generalization of minimizing an the mean absolute error (or I guess Laplacian MLE would be an equivalent way of saying it). I'm debating first on what ...
-1
votes
0answers
15 views
In optimization, what is the purpose of Gateaux and Frechet derivatives?
When reading Optimization related papers, I sometimes come across terms such as "Gateaux" or "Frechet" derivatives.
I'm often puzzled as to why we need these definitions, when we already know the ...
