In statistics this refers to selecting an estimator of a parameter by maximizing or minimizing some function of the data. One very common example is choosing an estimator which maximizes the joint density (or mass function) of the observed data referred to as Maximum Likelihood Estimation (MLE).

learn more… | top users | synonyms (2)

0
votes
1answer
18 views

Negative BIC in k-means

Probably a simple question but I'm trying to interpret BIC for k-means. I have some k-means clustering and calculating BIC gives me a negative value, with a plot something like this: ...
0
votes
0answers
37 views

Classification and convex optimization of the cost function

From the literature I read that For a neural network, the cost function, J(W,b) is a non-convex function, gradient descent is susceptible to local optima; however, in practice gradient descent ...
1
vote
0answers
20 views

Inverse covariance estimation for generalised Lasso

I have implemented Lasso for estimating sparse inverse covariance case using ADMM in Matlab. Inverse covariance estimation using LASSO regularisation, X is the estimate, S is empirical covariance ...
5
votes
2answers
63 views

What is energy minimization in machine learning?

I was reading about optimization for an ill-posed problem in computer vision and came across the explanation below about optimization on Wikipedia. What I don't understand is, why do they call this ...
4
votes
1answer
66 views
+50

When does pam (partition around medoids) fails to find the optimal solution? (counter example?!)

If I understand correctly, the pam algorithm is a greedy search for a set of medoids such that no other set offers a lower cost (i.e.: the sum of distances of points to their nearest medoid). ...
0
votes
0answers
6 views

Reasonableness of a “fit” with more free parameters than fit targets?

I have a (physics) model with 19 free parameters, but only 12 experimental values (= fit targets) that need to be reproduced by the model. This be more precise: 19 parameters are put into ...
3
votes
0answers
21 views

practical implementation detail of Bayesian Optimization

I'm giving Bayesian Optimization a go, following Snoek, Larochelle, and Adams [http://arxiv.org/pdf/1206.2944.pdf], using GPML [http://www.gaussianprocess.org/gpml/code/matlab/doc/]. I've implemented ...
6
votes
0answers
80 views
+100

Advantages of Particle Swarm Optimization over Bayesian Optimization for hyperparameter tuning?

There's substantial contemporary research on Bayesian Optimization (1) for tuning ML hyperparameters. The driving motivation here is that a minimal number of data points are required to make informed ...
0
votes
0answers
20 views

What is the best way to optimize API implementation order? [closed]

I am working on an emulator and would like to build API support in the most beneficial order. Assuming I have a scored list of applications and know which APIs they each use, what is the best way to ...
0
votes
0answers
10 views

Intuition for Normalized Squared Loss error function?

In terms of optimization squared loss is perhaps the most common error function used for regression. I've seen another function named "Normalized Squared Loss" mentioned, described as The ...
1
vote
0answers
8 views

True or false? “sum of an m-strongly convex and a convex function is m-strongly convex” [migrated]

I would like to know if the following conjecture is true or false? If $f(x) = g(x) + h(x)$ where $g$ is m-strongly convex and $h$ is convex, then $f$ is m-strongly convex. NOTE: For a ...
1
vote
1answer
26 views

Sample Representing a Different Population

I have two sets of populations: containing 1.5 million and 5.5 million units. I need to select a sample out of 5.5 million population so that the sample represents the 1.5 million population based on ...
0
votes
0answers
4 views

Looking for algorithm that is a discounted min-cost-maximum-flow calculation

In terms of graph theory I am very familiar with minimum-cost maximum flow, connectivity and ...
2
votes
1answer
35 views

How to force exponential regression to go through an intercept? [closed]

Excel has a handy function that lets you set the y-intercept of an exponential regression model: How can the same effect be achieved using R? I'm using the following code: ...
3
votes
1answer
19 views

What is the purpose of +1/-1 constraint on SVMs

It is common knowledge that all SVMS try to optimize some permutation of the following equation with the provided constraints: however, my question lies on the purpose of the constraint >= 1 and ...
0
votes
0answers
12 views

Choice of Boltzmann Exponent for Simulated Annealing

I was wondering if there was a good source on the optimiality of different choices for the "Boltzmann Exponent" for Simulated Annealing. Mathematica defines it here, though it would generally apply to ...
1
vote
0answers
12 views

Finding optimal values of parameters using observations

I have a problem that at first seemed super easy to solve but right now I am not sure how to crunch it. I have data with multiple observations (about 30) of certain process. Process can be modulated ...
0
votes
1answer
10 views

Is there a framework for reinforcement learning with states and actions in the same domain?

In reinforcement learning, there are states, actions, initial states, terminal states, a progress function and a reward function. Is there a theoretical framework or setting where states and actions ...
0
votes
0answers
5 views

Quantization threshold selection

I have the 256-bin histogram representing a distribution of the values taken by a certain descriptor element. This descriptor element takes the values in 0-255 range, hence 256 bins. I want to ...
1
vote
0answers
36 views

Looking for canonical problems or seminal work it the intersection of constraint programming and statistics

I'm interested in exploring the area (if it exists) at the intersection of constraint programming and statistics. My primary interest is on problems that require a combination of both statistical and ...
0
votes
0answers
5 views

Splitting of parameter space when using random mutation hill climbing

I have developed an agent-based model in NetLogo, and to calibrate the model and its parameters I want to use the random mutation hill climbing method. However, since my model is computationally ...
1
vote
0answers
19 views

Choosing the good initial value of the Newton-Raphson iteration method for Maximum Likelihood Estimation

I want to estimate the four parameters of Exponentiated Modified Weibull Extension (EMWE) distribution introduced by Sarhan and Apaloo (2013) with the Maximum Likelihood Estimation. Because the first ...
0
votes
0answers
10 views

Improving convergence of simulated annealing

I have a complex curve that I need to approximate with a parametric model (5 parameters). The parametric model itself is highly non-linear, and small changes in parameters can lead to big changes in ...
0
votes
0answers
12 views

Optimizing Graph to show areas of improvement

I have a data set that looks like below Bucket: 60-65%, 65-70%, 70-75%...... Days: 22 55 21 Bucket is a % performance and days is the number of days ...
2
votes
1answer
28 views

Justification of simulated annealing versus random search

I have a set of 16 integer parameters to optimize. The parameter space is too big for an exhaustive search, so I am using simulated annealing instead. I think my simulated annealing works - it finds ...
0
votes
0answers
12 views

optimization problem rewrite help

From my understanding, the transformation between probability and information content for d=2 is as follows: Information = -log_2 (Probability) If I have an optimization problem that is written in ...
0
votes
1answer
49 views

Optimization with both L1 and L2 regularization

After doing some research I suppose the hard part is that, L2 regularized problem is often solved by gradient descent, while L1 regularized problem is often solved by coordinate descent. But which ...
1
vote
0answers
23 views

Do I distort information stored in covariance matrix if I normalize the matrix to range of $[-1 , 1]$

I am using covariance matrix to maximize mutual information between samples that I selected and sample that I did not selected. The optimization algorithm that I use is sfo_greedy_lazy. It computes ...
1
vote
1answer
34 views

linear kernel SVM

The linear kernel is defined as: $K(x1,x2)=\langle x1,x2\rangle$. I can see that all that this kernel does is to calculate the dot product in the original space of the data. Why is this kernel then ...
0
votes
0answers
29 views

Error in arima optim

I have the following dataset. I tried arima with xreg ...
1
vote
1answer
49 views

BFGS & LBFGS for linear regression (overkill or compatibility issue)

BFGS and LBFGS algorithms are often seen used as optimization methods for non-linear machine learning problems such as with neural networks back propagation and logistic regression. My question is ...
0
votes
0answers
52 views

How can I prove that the log-likelihood function for logistic regression is globally concave?

For my master thesis, I have to show/prove that the log-likelihood function for logistic regression is globally concave. My supervisor told me that one way to show this is to use the fact that $X'X$ ...
1
vote
0answers
37 views

What is the correct equation of AdaGard one should use if one aims to use AdaGrad in practice as the automatic way to choose the step size?

I was reading Duchi et al. AdaGrad paper and also a shorter paper on Adaptive Online Gradient Descent because I was looking to implement an update rule which was automatically chosen and that was ...
0
votes
0answers
4 views

Optimization with solnp in R [migrated]

I am trying to do a simple optimization in R and I am confused at the moment, because of the following error. ...
1
vote
1answer
124 views

minimizing total travel time [closed]

Green circles are salesman and A-B-C are destination points. I need to move green circles so that total travelled time should be minimum and each salesman should go to a different point. So ideal ...
1
vote
0answers
25 views

Help: Random Forest optimization (image classification)

I'm having trouble classifying images using a random forest. The images all have a very similar scale, but they may be rotated arbitrarily around a fixed point in the image. The core problem is ...
2
votes
0answers
23 views

How to determine the number of random initializations to use in non-metric multidimensional scaling?

I'm trying to determine how many random initializations (restarts) I should use when performing an nMDS ordination. I understand I want to choose the solution that minimizes the stress, but how many ...
0
votes
1answer
23 views

Can I run several independent samples test to compare two stochastic algorithms?

I want to compare performance of two stochastic optimization algorithms: A1 and A2. Performance is defined as the output of a run. The lower the output is, the better is the solution the algorithm ...
1
vote
0answers
23 views

Data's Effect on Optimization Algorithm running into a Saddle Point

I am working with a non-linear model that uses seven independent variables to estimate a Bernoulli probability. To estimate the parameters of the model, I am optimizing a likelihood function using ...
4
votes
1answer
57 views

Ill-conditioned covariance matrix in GP regression for Bayesian optimization

Background and problem I am using Gaussian Processes (GP) for regression and subsequent Bayesian optimization (BO). For regression I use the gpml package for MATLAB with several custom-made ...
0
votes
0answers
9 views

What is the role of the coordinator in DistBelief's Sandblaster L-BFGS?

I'm having trouble wrapping my head around the Sandblaster L-BFGS. How often is the entire parameter vector in the same shared memory? What exactly is the purpose of the parameter server? And the ...
2
votes
2answers
70 views

Finding the Peak of a Kernel Density Estimator

I implemented a Kernel Density Estimator. I have a multivariate dataset that I use with it and I would like to find the point of highest likelihood. A way I thought about is sampling n points using ...
1
vote
1answer
28 views

Optimal output from continuous inputs

I have data consisting of a continuous dependent variable and several continuous independent variables. I need to figure out the optimal values of the independent variables that will maximize the ...
3
votes
1answer
58 views

Mathematical foundation of using MCMC in global optimization

MCMC is commonly used to compute the integral in the form of $$\text{Problem A.}~~\int F(x)\pi(x) $$ where $\pi$ is hidden. In the literature, it is explained why MCMC can handle problem A by ...
1
vote
1answer
64 views

minimum cdf given mean and variance

Suppose I have some distribution with mean $\mu$ and variance $\sigma^2$. How to prove that for the distribution to have min $cdf$ over all the distributions with mean $\mu$ and variance $\sigma^2$, ...
0
votes
0answers
36 views

How much does adding few input features to a SVM model affect its 'optimal' parameters?

I have trained a SVM model with RBF kernel, the parameter values of which have been selected by grid searching a wide range of (cost, ...
9
votes
4answers
373 views

Why do smaller weights result in simpler models in regularization?

I completed Andrew Ng's Machine Learning course around a year ago, and am now writing my High School Math exploration on the workings of Logistic Regression and techniques to optimize on performance. ...
4
votes
1answer
49 views

My ReLU network fails to launch

So I have a problem. Simple situation: Fully-connected Multi-Layer Perceptron with Rectified Linear (ReLU) units (both hidden and output layers), 1 hidden layer of 100 hidden units, trained with ...
3
votes
1answer
56 views

What types of functions can be implemented in a layer of a Neural Networks?

One of the most common algorithms for training Neural Networks is back propagation, which essentially does (stochastic) gradient descent on the training objective function. Gradient descent can be ...
0
votes
0answers
13 views

Modeling Market Impact in QCQP Form

I am trying to recast an optimization problem for portfolio selection including a 3/2 market impact term to a standard QCQP form. In particular, my initial optimization can be stated as wanting to ...