Tagged Questions
Use this tag for any use of optimization within statistics.
0
votes
0answers
8 views
If we “delete” the term of norm in SVR problem formulation, can it be solved with simplex method?
The problem formulation in Support Vector Regression is,
What if we don't want to take the "flatness" term, i.e., $\frac 1 2 ||w||^2$ and delete it; can we find solution from simplex method for ...
0
votes
0answers
12 views
Optimal threshold for rejecting in classification model
Let's say I have a model to detect fake product. The model predict the probability whether the product is fake, with 0.0 being authentic, 1.0 is fake. Each true prediction (that product is not fake), ...
3
votes
0answers
47 views
Maximum likelihood: Why is the number of non-zero eigenvalues equal to $x^T \hat{\Sigma}^{-1} x$
I've been reading this code (based on this R package) and I found that the number of non-zero eigenvalues of the estimated covariance is roughly equal to $x_i^T \hat{\Sigma}^{-1} x_i$. I want to know ...
0
votes
0answers
16 views
What are good online course/s for learning optimization from beginning to advanced (linear, convex, integer, geometric, nonlinear, ..etc)?
I need some online course or group of courses, preferably videos, that allow me to understand optimization from the beginning. Give me a solid base knowledge (what is linear programing, what is convex,...
1
vote
0answers
15 views
DeepMind RMSprop vs. Tensorflow RMSprop
Some neural network architectures work better with RMSprop than e.g. ADAM. So for example stated by DeepMind in their work with Atari games and reinforcement learning.
Maciej Jaskowski reproduced the ...
0
votes
0answers
13 views
The miracle of the Lanczos/conjugate gradient algorithm
Lanczos/Arnoldi/Rietz/CG-like algorithm share the same core strategy... In each, a little miracle appears, most of the Gram-Schmidt inner products are zeroes ! In others words, new direction need only ...
2
votes
1answer
24 views
Backpropagation wrong? Doesn't it update dependent variables in hidden layer
In a multi layer perceptron or feedforward neural network, isn't backpropagation updating weights of the middle layers that are dependent variables? So for a particular hidden layer, it calculates all ...
0
votes
0answers
9 views
Estimation of infinitesimal generator/transition rate matrix from proportion data
Suppose I have a collection of data $\{\boldsymbol x_t \in \mathbb S^d\}_{t = 1,\dots,T}$ where $\mathbb S^d$ is the $d$-dimensional unit simplex, i.e. the elements of $\boldsymbol x_t$ sum to $1$.
...
3
votes
1answer
84 views
+50
Optimizing $\chi^2$ using MCMC
I have measurements of an object.
Let's say I have its length $L$, mass $M$, and age $t$: $$\mathbf y = (10~\text{m},\ 0.01~\text{g},\ 5~\text{s}).$$ I also have the uncertainties on my measurements ...
0
votes
0answers
15 views
Fast constrained optimization algorithm
I have to estimate parameters which have to be greater than zero. First I tried L-BFGS-B, but with different starting values I'm getting different results everytime. Now I switched to differential ...
1
vote
0answers
15 views
nlme ignoring certain control arguments to nlminb
Issue: Do constrained optimization of parameters in nlme::nlme
I'm trying to fit a non-linear mixed effects model using nlme::nlme, which can use 2 optimization schemes: stats::nlminb or stats::nlm.
...
0
votes
0answers
18 views
which parameter you choose on lasso CV, tuning parameter λ or βi constraint s?
I try to use lasso for prediction and I have $X_{tr} \subset X$ the train set and $Y_{tr}$ the train target.
and I have $X_{ts} \subset X$ and $ Y_{ts}$ the test set for CV.
I used CV and got $λ_i$ ...
0
votes
0answers
8 views
Fitting data on a time demanding stochastic Model
I have a multi-parameter (8 at least) model which is very time consuming. It's not an analytical function but instead is a model which integrates many differential equation and some times the result ...
0
votes
0answers
7 views
Parameter setting in minitab Question
I want to optimize parameters of a genetic algorithm in Minitab. I want to use Taguchi method. To do this, I created three random datasets (with similar seeds) and run each one five times. and ...
1
vote
2answers
44 views
Why is the second derivative required for newton's method for back-propagation?
I am troubled with why isn't the Newton's method used for backpropagation, instead, or in addition to Gradient Descent more widely.
I have seen this same question, and the widely accepted answer ...
1
vote
0answers
18 views
Mixed integer programming R: Least absolute deviation with cost associated with each regressor
I have been presented with a problem, regarding the minimization of the absolute error, the problem know as LAD(Least absolute deviation) but, being each regressor the result of expenssive test with ...
0
votes
0answers
10 views
scipy L-BFGS-B optimizer with different step size per dimension [migrated]
How can I adjust the optimizer to use a different step size for each DOF? When I print the parameters the step size seems to the the same per dimension.
Any other alternative optimizer that can ...
0
votes
1answer
3 views
Is there any guideline of when to use IBEA and NSGAII?
I am aware that IBEA and NSGAII are using different metrics for evolution and mutation, is there any guideline on when to use each of them?
0
votes
1answer
61 views
Feasibility of a neural network fitting a specific multivariate quadratic function? [duplicate]
I have run into some problems when trying to train a network that fits some multivariate quadratic function, or the Euclidean distance between 2 points in a 3-dimensional space, where they are 'pretty ...
1
vote
0answers
45 views
How to solve an optimization problem with variable in indicator function?
How to solve the following optimization problem?
$$
\underset{D, U \in \mathbb{R}}{\min} (1+a)\text{E}_{X}[(X-D)\cdot\mathbb{1}_{\{D<X\leq U\}}] - (M-D)\cdot\mathbb{1}_{\{D<M\leq U\}},
$$
where ...
0
votes
0answers
11 views
Optimising for a set number of positive
I am working on a fraud detection use case for loan applications.
Thousands of loans applications are made every day but the ressources to check whether the positive fraud cases are true or not are ...
0
votes
0answers
15 views
Optimization of cost function using newton method
I am studying "Distributed newton method for deep neural network", a research paper by CC Wang and his associates.
In the section of "Non-hessian Newton method", there is a mention of second-order ...
0
votes
0answers
14 views
Error message when using nlminb in R [migrated]
When I tried to use nlminb to optimize a function with three parameters, it showed this error message
...
0
votes
0answers
48 views
how to understand the process of handle the missing feature with em algorithm?
I just learn the book 'pattern classification (Richard O. Duda, Peter E. Hart and David G. Stork, 2nd ed., pp. 32-33)' and there's an example to handle the missing feature with em algorithm. I found I ...
1
vote
1answer
68 views
Regularizing the inverse coefficient matrix in multivariate regression
I'd like to minimize the objective
$ \operatorname{tr}[ (Y-XR^{-1})^T (Y-XR^{-1}) ] + \lambda \sum_{ij} |R_{ij}|$
wrt to $R$ (which is $P \times P$ but non-symmetric) where $Y$ and $X$ are both $N \...
0
votes
0answers
13 views
Scikit-learn, cross-validation with parameter optimization
I wrote this idea in Python, but I'm confused as to the outcome of it.
I'm trying to do parameter tuning, with cross-validation. To do this, I have a classifier, a model selection strategy (...
1
vote
1answer
20 views
How do I best use a fit statistic like chi-squared fit for a model that predicts two independent sets of measurements?
I have a model $M(\vec{x})$ for a vector of model parameters $\vec{x}$ that predicts two sets of measurements that I have taken - $v(h)$ and $L(h)$. The two independent data sets each have their own ...
0
votes
0answers
10 views
GPy Opt : Optimize acquisition function over a subset of input variables
I'm trying to use GPyOpt library from Sheffield for an application of Bayesian Optimization. Specifically, currently I'm working on how to make a new acquisition function in the application domain ...
1
vote
0answers
10 views
Segmentation Visualisation DNN
I was interested in what the network is exactly learning during the process of image segmentation. Take unet for example. In image classification, the hierarchy of features is more clearer to ...
1
vote
0answers
11 views
Any previous work on online coordinate descent, where a new coordinate appears at each iteration?
I'd like to analyze the behavior of coordinate descent algorithms, where as a twist, at each iteration, a new variable appears. For example, if $T$ is my total number of iterations, then at iteration $...
0
votes
0answers
25 views
Unable to solve using lagrangian multipliers
Suppose
$$K(x,z) = \theta(x)^T \theta(z) = \left\{ \begin{array}{ll}
1 & \text{if } x = z \\
0 & \text{otherwise} \end{array} \right.
$$
and $y_1=+1$ or $-1$.
I ...
1
vote
0answers
28 views
Estimation Multinomial Logit [closed]
I need to create a code manually corresponding to the likelihood of the multinomial logit model in R. I have not been able to get the same results from some packages (mlogit, multinom).
My database ...
0
votes
0answers
18 views
K means minimize Sum of SSEs of clusters [duplicate]
I am talking about K-means clustering problem.
Theoretically, my understanding is :
If there are K clusters, the K-means algorithm outputs K centroids/means such that it minimizes total sum of SSE ...
0
votes
0answers
17 views
How to optimize two related objective functions?
Suppose we have two functions which are somewhat related in that if one increases, the other decreases in a non-linear fashion. The decision variables for one function may or may not be present for ...
1
vote
0answers
30 views
Why the sign is *plus* in neural network [closed]
REFERENCE
GITHUB GIST
I wanted to implement Neural Network with Numpy in Python. Then I have two question. The first one is about the sign
...
0
votes
0answers
19 views
Is it possible to combine SPSA and Adam?
In SGD algorithms such as Adam you generally make a bad estimate of the gradient of the loss function and take that gradient to move the parameters in the desired direction.
Gradient free methods ...
9
votes
4answers
1k views
Gradient descent optimization
I am trying to understand gradient descent optimization in ML(machine learning) algorithms. I understand that there's a cost function—where the aim is to minimize the error $\hat y-y$. In a ...
0
votes
0answers
42 views
Reproducing Adam paper with scikit-learn
I am trying to reproduce the results of the article introducing the algorithm Adam using scikit-learn. More specifically, I want to reproduce figure 2b, where the authors monitor the evolution of the ...
1
vote
1answer
50 views
using regression model to optimize teams working on work items
I have a few work items with these features:
WI1, WI2, WI3
which describe these work items. I also know the number of people and how many minutes they spend each ...
1
vote
0answers
24 views
Question about Geometric Margin of Support Vector Machine
I'm trying to follow Andrew Ng's notes on Support Vector Machines and had the following question.
In his notes, Ng, transforms the following optimization problem [using the notion of geometric ...
2
votes
0answers
34 views
How many sunrises are worth observing?
The one-sun version of Laplace's sunrise problem provides a Bayesian argument that, if on all $n$ mornings in recorded history the Sun has risen, its probability of doing so tomorrow is $\frac{n+1}{n+...
0
votes
0answers
92 views
Kalman Filter Parameter Estimation of Yield Curve Model
I'm trying to reproduce in R with DLM or KFAS package the results from Diebold, Rudebusch, Aruoba paper The macroeconomy and the yield curve: a dynamic latent factor approach with some help from ...
0
votes
0answers
11 views
Solve an equation of two unknowns to fit a distribution and mean
Apologize in advance for the drawn out question (found at the end). But I want to give a comprehensive picture of the problem.
I have an equation of the following form:
$T_{ij}^{sim}=A_i*O_i^{^{obs}...
4
votes
3answers
105 views
Are there any supervised learning methods that do NOT boil down to optimizing a loss function?
All of supervised learning methods I can think of amount to optimizing a loss function (RMSE, AIC, Cross-Entropy,...) against a labeled data set. One would think that "learning = optimizing loss ...
1
vote
0answers
21 views
How to diagnose why numeric solver is not converging? [closed]
Are there specific approaches, methods or software that help in determening why an optimum is not found for a particular optimization problem?
For example, the solution landscape could be visualized ...
2
votes
0answers
23 views
Optimization textbooks for statistics and data analytics
Any statistical analysis, machine learning or data science involves some sort of optimization at the end of the day.
I'm looking for good linear and nonlinear optimization textbooks for self ...
2
votes
1answer
37 views
Does optimizing the parameters in an exponential smoothing model constitute “learning”?
I'm having an argument at work with a colleague who's saying that we need to use machine learning models instead of the current exponential smoothing models (Holt, Holt-Winters) for demand forecasting,...
0
votes
0answers
7 views
Regression with Convex Quadratic Maximization over a polytope
I am doing research on "optimization for machine learning & statistics". Since I have a strong focus on optimization, I am quite curious about the problem classes we have in regression problems.
...
0
votes
0answers
8 views
Using the concept of commodity flow, how do I check if a sub-graph is connected for MIP optimization problems?
The figure below shows two graphs, one is connected, one is not. Using network flow concept, I have set Node 1 as a source to send one unit of flow to each other node. For connected graph, there ...
0
votes
0answers
5 views
What hyperparameters should be sampled (together) for neural networks?
I'm using a neural network for a multi-target regression task and would like to perform hyper-parameter optimization. The network has one hidden layer and uses MSE loss on the output. I have large ...
