In statistics this refers to selecting an estimator of a parameter by maximizing or minimizing some function of the data. One very common example is choosing an estimator which maximizes the joint density (or mass function) of the observed data referred to as Maximum Likelihood Estimation (MLE).

learn more… | top users | synonyms (2)

0
votes
0answers
20 views

Quasi-Newton Accelerator (QN1) for EM Algorithm

I am trying to implement what is called a "very simple to implement" accelerator for the EM algorithm. Specifically I am talking about the QN1 algorithm, described here, and am using a multivariate ...
0
votes
0answers
11 views

Non - Linear, Non-Smooth Optimization in Excel [migrated]

I am trying to solve a non-linear, non-smooth optimization in excel. Both GRG and Evolutionary algorithms are not able to give reasonable results(they are not converging in certain cases). The number ...
1
vote
2answers
32 views

How to use an optimization solver to get t-stats and p-values for the estimates?

I calculate a data log likelihood (evaluated at a set of parameters to be estimated), and my task is to find the set of parameters that maximize my log likelihood. My problem is: thought there are a ...
0
votes
0answers
7 views

Heuristics for streaming data matching [duplicate]

I have an index composed by thousands of documents. Slightly modified copies of those documents are sent to my application in small chunks, and I need to check, from those chunks, which document has ...
3
votes
0answers
24 views

Heuristics streaming data matching

I have an index composed by thousands of documents. Slightly modified copies of those documents are sent to my application in small chunks, and I need to check, from those chunks, which document has ...
0
votes
0answers
21 views

Checking error in optimisation problem

My apologies if this problem is trivial -- I do not have much experience with statistical methods! Hopefully someone can point me in the right direction. Background: I am currently working on an ...
0
votes
1answer
26 views

Online gradient descent for strongly convex function

Given that our loss function is $\alpha$ strongly convex function which means $\mbox(\nabla f(\mathbf{x})-\nabla f(\mathbf{y}))^{T}(\mathbf{x}-\mathbf{y})\geq \alpha||\mathbf{x}-\mathbf{y}||_{2}^{2} ...
4
votes
1answer
69 views
+50

Is an SVM's (maximum) likelihood uniquely defined as a function of hyperparameters?

I think that I must be reading this paragraph (below) incorrectly. Note that both types of evidence that we have defined in general depend on the inverse noise level $C$ and the kernel $K(x, ...
0
votes
1answer
23 views

Coordinate vs. gradient descent

I was wondering what the different use cases are for the two algorithms, Coordinate Descent and Gradient Descent. I know that coordinate descent has problems with non-smooth functions but it is used ...
1
vote
0answers
15 views

Extensions to distributed Stochastic Gradient Descent

In the conclusions of the paper for distributed Stochastic Gradient Descent using mini-batches the authors discuss a problem of the proposed method regarding the synchrony requirements. Specifically, ...
0
votes
0answers
24 views

How do you determine neural network loss function when there are multiple outputs?

This great Youtube tutorial taught me how to fit a neural network with one output. To apply back-propagation, you first find the Jacobian of the loss function with respect to the weights. ...
0
votes
1answer
24 views

For a quadratic form to minimize with a L2 regularization term, is the gradient of the solution collinear to the solution?

Say you minimize a quadratic form f with a L2 regularization term (g = f + L2_term). The solution of minimizing g is x*. Is the gradient of f applied to x* collinear to x* as the figure below ...
0
votes
0answers
6 views

Worried about hidden costs of using this loss function to fit weights

I have the following model: $$\frac{Y}{T} = f(X \beta)$$ where beta is a vector of weights, and Y and T - Y are greater than 0. I want to fit the $\beta$ vector using the loss function $$ ...
1
vote
0answers
13 views

Expectation Propagation - Computing mean and variance of error function

I'm still trying to wrap my head around computing the moments for the expectation propagation algorithm and whether I can use it for the following example: say i have a product of distributions which ...
0
votes
1answer
57 views

Neural networks: how can convex optimization produce different weights each time?

I am training a multilayer perceptron with a logistic activation function by backpropagation. The weights are not unique - each time I redo the fit, I get a different set of weights. However the ...
2
votes
1answer
37 views

L-BFGS-B converging in iteration 0 in R [closed]

I am trying to fit an F distribution to a given set using optim's L-BFGS-B method. For some reason, it is always converging at iteration 0, which obviously doesn't ...
1
vote
1answer
65 views

Advantages of taking the logarithm to minimize the likelihood

In regression/classification problem, we are often interested in minimizing a cost function with respect to the parameters of the model. In many cases, the cost function is the negative likelihood. To ...
2
votes
0answers
28 views

piecewise linear regression with unknown number of knots

I have a model that depends linearly on $v$ and $\alpha$, but not linearly on other two parameters $T_0$ and $T_1$: $f(i; v, \alpha, T_0, T_1)$. Using least squares, I can solve for $v$ and $\alpha$, ...
0
votes
2answers
34 views

Normalized gradients in Steepest descent algorithm

In general setting of steepest descent algorithm we have, \begin{equation} x_{n+1}=x_n-\alpha G_n, \end{equation} where $\alpha$ is the step size and $G_n$ is the gradient evaluated at the point ...
8
votes
3answers
604 views

Why are second-order derivatives useful in convex optimization?

I guess this is a basic question and it has to do with the direction of the gradient itself, but I'm looking for examples where 2nd order methods (e.g. BFGS) are more effective than simple gradient ...
1
vote
0answers
31 views

How do you do EM algorithm for a factored model for a recommender system?

Let $X$ be a $n \times d$ matrix with users as rows and movies as columns. Each user is a single row $x^{(u)} \in \mathbb{R}^d$ (i.e. for user u there are at most d ratings for the d movies). Also ...
0
votes
0answers
27 views

Obtaining Hopfield network weights from energy function

I'm trying to come up to speed on Hopfield networks. When applied to solving the traveling salesman problem, for example, papers present their energy function, but never explain how they derive the ...
0
votes
0answers
16 views

Minimizing the sum of a smooth function and a singular one?

I'm interested in minimizing a smooth function: $ f(p,q)= f_1 +f_2 $, where $f_1$ is smooth and I know its gradient, but $f_2$ is not differentiable, in fact $f_2(v):=a(|v_1|+|v_2|)$, for some scalar ...
0
votes
0answers
40 views

How do I sample from a black-box model of a probability distribution?

I have a function 'P(x)' where we query for any 'x' it gives a probability value. This function 'P' does not have a closed form and the evaluation is costly. Now 'x' is a set of vectors(matrix whose ...
6
votes
1answer
112 views

Jenks Natural Breaks in Python: How to find the optimum number of breaks?

I found this Python implementation of the Jenks Natural Breaks algorithm and I could make it run on my Windows 7 machine. It is pretty fast and it finds the breaks in few time, considering the size of ...
1
vote
0answers
12 views

Compare optimization algorithms with stop criteria

I'm comparing Harmony Search and Genetic Algorithm for a specific finance case study. I set a early stopping criteria. If we don't have improvement in 1/5 of total iterations of algorithm, It will ...
3
votes
1answer
51 views

Maximizing combined log likelihood from different dataset using r

I have observed data $y$. And I have a function that gives me estimated $\hat{y} = f(x,\hat{P})$ where P is the parameter I want to estimate. I was able to optim() command in R to get maximized ...
0
votes
1answer
21 views

Non linear optimization with objective function as a string [closed]

I am looking for a package to help me solve some non linear optimisation problems with constraints. The objective function f to optimize is given as a string. For example, f is given as the following ...
0
votes
0answers
41 views

Are there ways to improve Levenberg-Marquardt backpropogation performance in Neural Networks?

When using Levenberg-Marquardt optimization for a function approximation problem, the performance and speed generally trumps that of the gradient descent. I am approximating the functions cos(n * Pi) ...
1
vote
2answers
30 views

Recommended/estimated number of radial basis functions in RBFN

thank you for taking the time to read my question. I am attempting to make a Radial Basis Function Network to see if a relationship exists between input/output data that I have been collecting. I ...
1
vote
0answers
9 views

How can I find the bounds that gets Simulated Annealing to converge?

According to Wikipedia on Simulated Annealing, For any given finite problem, the probability that the simulated annealing algorithm terminates with a global optimal solution approaches 1 as the ...
1
vote
1answer
19 views

Confidence that Optimum Lies Within a Given Region

I have an ordinary least squares regression equation. Through calculus or simulation, I can find the combination of explanatory values that maximize the estimated mean response; I can also establish a ...
2
votes
3answers
105 views

Machine learning for discovering formulas

I have a vector of desired values that I want to fit based in some generated predictors. The tricky part is that I wish to have the explicit formula. For example, giving the below input I would like ...
1
vote
1answer
24 views

How to find $\arg\max$ of a neural network?

Let's say I have a neural network $f$ that takes input $\vec x \in \mathbb {R}^n$ and produces output $f(\vec x) \in \mathbb{R}$. How can I find $\hat x = \underset{\vec x}{\arg\max} \; f(\vec x)$?
0
votes
0answers
18 views

Online EM?? Did any one implement online EM using matlab?

In Online EM algorithm in each iteration for each sample one estimates sufficient statistics and adds it to overall sufficient statistics and in maximization step one maximizes parameters to MLE for ...
4
votes
2answers
67 views

Expectation maxmisation algorithm increases true likelihood at each iteration

I've heard that the EM algorithm ensures that the true likelihood is non-decreasing at each iteration of the algorithm, but I'm not sure why this is the case. I've provided a basic plot which I ...
3
votes
1answer
67 views

What are the differences between various R quadratic programming solvers?

I am looking for a package to help me solve some quadratic optimisation problems and I see there are at least half a dozen different packages. According to this page: QP (Quadratic programming, ...
3
votes
1answer
29 views

What is the solution to this minimization problem?

I'm encountering the following minimization problem in my research: $$\hat b = \underset{b}{\arg\min} \sum_i^n \left( \log \frac{a_i}{b} \right)^2$$ I could iteratively optimize, but I think that ...
1
vote
0answers
17 views

Under what assumptions can parameter estimate uncertainty be estimated from the Hessian?

Given a model with some parameters, some data it's attempting to reproduce, and a distance function to quantify how well the predictions correspond to the data, I can fit parameters via a general ...
0
votes
1answer
16 views

Total Variation Denoising help

I am trying to work through the "Mathematical exposition for 1D digital signals" in the wikipedia entry for Total Variation Denoising (TVD). I am familiar with Lagrange multipliers. However, I cant ...
1
vote
0answers
14 views

Likelihood convexification

I am doing constrained vector optimization using a non-convex non-linear likelihood function. My problem is of the following form: $$\begin{align*}\hat Q &= \underset{\vec Q}{\arg\min} -\log ...
0
votes
0answers
20 views

Implementing evolutionary algorithms

We're trying to to minimize the following functions using Multi Objective Evolutionary Algorithms : $\mathrm{minimize}~\lambda_q(1-a)E(N)E(X)+\lambda_xE(N)E(\max(X/a-R,0))$ ...
0
votes
1answer
54 views

Clarification about Perceptron Rule vs. Gradient Descent vs. Stochastic Gradient Descent implementation

I experimented a little bit with different Perceptron implementations and want to make sure if I understand the "iterations" correctly. Rosenblatt's original perceptron rule As far as I understand, ...
0
votes
0answers
24 views

What is Tau and Omega in the Black Litterman model?

I'm looking into the BL model when it comes to portfolio optimization, and I'm having a hard time trying to understand each one. I've read on several papers that Omega is the covariance matrix, but I ...
0
votes
2answers
123 views

R optim function - Setting constraints for individual parameters

I am looking to minimize a function using optim as follows: ...
0
votes
1answer
71 views

R - Constrained optimization for a function taking a matrix

I would like to do constrained optimization for a function which takes a matrix as input. The matrix is sparse, representing a weighted adjacency matrix , and only the weights shall be subject to ...
2
votes
1answer
154 views

Constrained assignment problem (Linear Programming, Genetic Algorithm, etc…)

I'm looking for advice on how I should approach a specific problem. I have about 1000 shops that I have to assign to about 20 different supply centers out of a possible 28, and I'm trying to pick the ...
1
vote
1answer
31 views

On the usage of numerical optimization technique to maximize log-likelihood

Literature and resources say that when the ML log-likelhood does not have a closed form expression, then we can use Newton-Raphson and other optimization techniques. My Question is: During estimation ...
0
votes
1answer
64 views

Selecting optimization algorithms

I am in the process of developing a multi-state model (using the msm package in R), and have been encountering the error message initial value in 'vmmin' is not finite This error occurs ...
0
votes
0answers
45 views

Choose the correct regression model

The relation of two measurement values x and y might be linear mod1 <- lm(x~y) ...