In statistics this refers to selecting an estimator of a parameter by maximizing or minimizing some function of the data. One very common example is choosing an estimator which maximizes the joint density (or mass function) of the observed data referred to as Maximum Likelihood Estimation (MLE).

learn more… | top users | synonyms (2)

0
votes
0answers
9 views

How does Shuffled-Complex-Evolution-Metropolis algorithm compare to other adaptive samplers (e.g. NUTS)?

I recently heard of the Shuffled-Complex-Evolution-Metropolis algorithm and am curious how it compares to other adaptive MCMC sampling algorithms. Unfortunately I am still learning about optimizing ...
1
vote
0answers
12 views

How to find the threshold that maximizes the F1 score?

I have a probabilistic, binary classifier. Is there any principled way to select the threshold that maximizes the F1 score? Currently I simply choose many different thresholds, apply them on some ...
1
vote
2answers
49 views

How to constrain cumulative Gaussian parameters so that the function will intersect one given point?

I am analyzing data from one study where participants had to choose (between two stimuli) the one with higher intensity. One way to look at the data is to fit the proportion of correct choices as a ...
0
votes
1answer
32 views

SVD from Matrix formulation to objective function

I'm writing the question to try to complete the circle after reading the 2 other questions on Cross Validated and the link on the third bullet point: What is the objective function of PCA? What ...
0
votes
0answers
7 views

How to find the optimal weights that minimizes the error variance when combining multiple datasets in R [on hold]

I am trying to combine 2 datasets in R using the optim function to compute the optimal weights. However, i am quite new to R and is having difficulty writing the objective function with the following ...
0
votes
1answer
37 views

How do you turn the output of a nnet neural network model into an equation?

Assuming the output of the above nnet feedforward model (nnetModel) is such that the following summary is produced: ...
0
votes
1answer
29 views

GLMER and Model is nearly unidentifiable: very large eigenvalue

I'm working on a logistic regression analysis using the lme4 package and function glmer. I built the following model: ...
0
votes
0answers
14 views

How to solve diet problem using genetic algorithm GA technique in R [closed]

I would like kindly to ask if anybody has an idea how to build a genetic algorithm GA model and to solve the diet problem with constraints in R? thank you and appreciate your contribution
0
votes
0answers
61 views

Is it possible to use genetic algorithms as a learning algorithm for artificial neural networks?

I've been using back-propagation to optimize neural networks without problems. But I've read in some books that genetic algorithms can be used to optimize an ANN. I want to know if its possible to use ...
0
votes
1answer
34 views

Is Representer theorem valid with constraints on coefficients?

By the representer theorem, we have that in a Reproducing Kernel Hilbert Space the function being learnt in a regularizer + loss function problem under some conditions, can be represented as $\sum_i ...
1
vote
0answers
31 views

Gradient of a sum of indicators

EDITED w.r.t. whuber's comment: Say I have a function $\mathbb R^n \rightarrow \mathbb R$: $$f(w_1,\ldots,w_n) = \frac{n^-\sum_{i\in I^-}w_ix_i}{n^+\sum_{i\in I^+}w_ix_i}$$ with fixed $x_i\in\mathbb ...
1
vote
0answers
40 views

How does one extend radial basis function (RBF) networks formally from regularization but with vector valued outputs?

I was reading the following paper on hyper & radial basis function (HBFs & RBFs) networks and also this one that kind of summarizes the first one and was trying to understand how to extend ...
1
vote
0answers
73 views

What are key differences between Theano (Python) and Torch (Lua) for deep learning? [closed]

Theano and Torch both supports GPU calculations. My question is whether Theano or Torch have significant differences in: performance ease of use (assuming one knows the programming language) libaray ...
5
votes
0answers
69 views

Why do Elo rating system use wrong update rule?

Elo rating system use a gradient descent minimization algorithm of the cross-entropy loss function between the expected and observed probability of an outcome in paired comparisons. We can write the ...
0
votes
0answers
9 views

Instances of this Gaussian variance model with non-convex log-likelihood

On page 4 of Thomas Minka's Beyond Newton's Method, he mentions the following Gaussian variance model with a non-convex maximum likelihood objective. What are some examples/instances of models where ...
0
votes
0answers
12 views

Interpolation of Data Value using Optimized Weighting of Its Features

I have a question regarding "Interpolation" / "Prediction" of a value. Assuming we have a data set $ { \left\{ \left( {x}_{i}, {y}_{i} \right) \right\}}_{i = 1}^{N} $ where $ {x}_{i} \in ...
0
votes
1answer
22 views

First Derivative / Maximisation Problem

It appears to be super easy, but I just cannot reconcile how Lambert et al. (2012) on page 8 come from (3) to (4) - please see screenshot! Particularly bothering: where does the "Lambda" in the ...
0
votes
0answers
216 views

Learning algorithm to define prediciton region to maximize score

The title may not too fitting but here is the problem. You have a sample of data with each data point being an x,y coordinate and a score. The goal is to create a loop which encloses the values as to ...
1
vote
0answers
36 views

R optim function : can it be used as a Solver [migrated]

I saw another question here on this forum, and saw that the Optim function was used as a kind of Solver instead of as an optimizer. Here is the link: ...
1
vote
1answer
33 views

Bayesian optimization or gradient descent?

When and why use Bayesian optimization, instead of gradient descent? Which one is better for which cases?
0
votes
1answer
16 views

Rationale for MCEM

From what I've read, the main advantage of the EM algorithm is that the expectation step can be expressed in closed form giving a deterministic answer and thus 0 variance. What's the rationale then ...
0
votes
0answers
6 views

How to refine learned parameters with new untrained data?

I am trying to solve the inverse problem of the linear quadratic regulator (LQR) control. The idea is to find the parameters that minimize the discrepancy between the available demonstration and the ...
0
votes
0answers
9 views

ContrOptim Function- Error in Argument

I'm trying to replicate the Excel Solver in R- which is basically a constraint optimization problem I'm trying to minimize the cost per action which is total spend/ total actions which equals to the ...
0
votes
2answers
43 views

optimizing over a set of symmetric matrices

I need to minimize a complicated loss function, $f\left(\Lambda\right)$ over a set of symmetric matrices, $S_{p}$ of dimension p, such that all the eigenvalues of $\Lambda \in \left[0,1\right]$. I ...
0
votes
0answers
8 views

Spearman correlation optmization

I need to compute Spearman correlation between two vectors. One vector is fixed, the other depends on k variables. I need to find set of variables {l1..lk} so that Spearman correlation is maximized. ...
0
votes
0answers
26 views

marginal likelihood in linear bayesian regression (in weight-space)

I want to tune the hyperparameters namely the target deviance $\sigma_y$ and weight deviance $\sigma_w$ in bayesian linear regression. The posterior distribution in level-1 inference which is ...
0
votes
2answers
94 views

Optimization of Complicated Function with Two Random Variables

I am trying to find $x$ that will minimize the following expression which involves two sources of randomness. I am stuck and not even sure where to start. Any suggestions would be appreciated. Please ...
1
vote
1answer
20 views

How to chose a store by optimizing my costs?

Let's say that I wanted to chose between two grocery stores (store $a$ and store $b$) in my area. They have the same items, and they both charge a variable price / cost for each product ($a_1$, $b_1$, ...
1
vote
0answers
15 views

How to optimize a generalized trace problem in dimensionality reduction

I know how to solve this problem in dimensionality reduction. $argmax_{X}$ $Trace[XLX^T]$ with $XX^T=I$ ,where $L$ is symmetric, $X$ is unitary, and $I$ is identity matrix. But I'd like to know how ...
1
vote
1answer
19 views

Solving “n” equations with 3 unknowns

I'm new to R and I'm trying to solve a system of equations. I have about 380 equations where i have 3 unknowns per equation. I can use three equations and solve by using "solve()" and it works great. ...
1
vote
1answer
26 views

Estimating error for parameters from multiple regression with linear constraints

I am working on a multiple linear regression problem where I would like to constrain only some of the parameters to non-negative values. There have been discussions of how to solve for the parameters ...
0
votes
0answers
14 views

when to stop this convex optimizations algorithm?

I am reading the article with title "metric learning with collaping classes" lately http://papers.nips.cc/paper/2947-metric-learning-by-collapsing-classes.pdf. See this thread (what is 1/0 in this ...
1
vote
1answer
97 views

Optimization of a Convex Function involving Standard Normal CDF and PDF

Could someone provide closed form solutions, if any, and steps to get there for the following optimization problem? Please note this function has been shown to be a convex function and hence a minimum ...
2
votes
1answer
46 views

How does one do Stochastic Gradient Descent (SGD) on an objective function that has a regularizer?

I know that for Stochastic Gradient Descent, one picks a data point $(x_n, y_n)$ at random from the training set $S_N$ and then updates the parameter of the model in question. If the cost function ...
1
vote
1answer
24 views

Why is it necessary to divide by the number of samples when optimizing squared error?

In a lot of different optimization problems, and with particular regard to gradient descent, we use the mean squared error as a loss function. In the formulation of mean squared error, you divide by ...
0
votes
0answers
20 views

How to demonstrate connection between performance and difference between real and optimal values of the parameters

I have the data from the experiment with 140 participants. I model the process of decision-making during the experiment and estimate the values of two parameters (A and B). Then, using simulations, I ...
1
vote
1answer
37 views

PSO Clustering using R using Repplab package

I wish to try clustering a matrix of numerical data using swarm intelligence. (Matrix is 28000 X 53 and sparse). I'm working in R and found the REPPlab package and used the EPPlab function. My ...
0
votes
0answers
86 views

Finding the optimal threshold parameter

Assume we are penalizing the least squares by the hard thresholding penalty: $argmin_\theta 2^{-1}(z_2-\theta)^2 + p_\lambda(|\theta|)$ where $p_\lambda(|\theta|)$ is the hard thresholding penalty ...
0
votes
0answers
21 views

How to optimize costly, smooth, multidimensional, varying scale function with flat regions and slight noise

I am trying to optimize hyperparameters of a complex model. Each iteration takes roughly 30s (during which the lower level model is run many times). I believe the underlying function to be generally ...
0
votes
0answers
18 views

What Technique to use to Optimize Future Prices?

I work for a company that sells different levels of ad packages on its website in various markets. These ad packages are sold contractually for specific periods of time (e.g. 3 months, 6 months, 1 ...
0
votes
0answers
22 views

What if we use mean and standard deviation in Stahel-Donoho outlier measure?

I need to use an outlyingness measure in an optimization problem which is already complex. So I need a simple measure of outlyingness. I didn't find any except Stahel-Donoho outlyingness measure. In ...
3
votes
1answer
30 views

Ensuring parameters of log linear model sum to 1

I am training a log-linear model with parameters $\theta$ using SGD. I want to ensure that my parameters will end up being probabilities i.e. $\sum_i \theta_i = 1$. One way to do this is by using ...
0
votes
1answer
63 views

Hessian for linear regression with regularization

I'm using matlab to solve a regularized linear regression via the fminunc() function. The cost function is from the standford machine learning class. It's pretty slow so and I think it could be sped ...
0
votes
0answers
16 views

What are some major theories on picking the right number for the sample window size in time series analysis?

for example the number of samples to run the moving average, or the number of samples for sequential hypothesis testing. Or if there is a control scheme going on what is the best time window for an ...
0
votes
0answers
24 views

Tikhonov regularizes least square dual function

I'd like to find the dual problem for least squares with Tikhonov regularization. For now, I have the primal problem expressed as minimize $||Ax-b||_2^2 + \gamma||x||_2^2$. I'm introducing a dummy ...
0
votes
1answer
125 views

How to minimize class weight vector of Random Forest Classifier using CV

What I'd like to do is optimize the class weights of a Random Forest Classifier (using python and the sklearn library) for multiclass classification, in which different misclassification errors have ...
1
vote
0answers
69 views

Hyperparameter tuning in Gaussian Process Regression

I am trying to tune the hyperparameters of the gaussian process regression algorithm I've implemented. I simply want to maximize the log marginal likelihood given by the formula ...
0
votes
0answers
32 views

Linear - Quadratic optimization for system of objectives

I have two distinct data sets, $\{x^{\mu},J^{\mu}\}$, $\mu=1,\ldots,n$ and $\{x^{\nu},V^{\nu}\}$, $\nu=1,\ldots,m$ that also include uncertainties $\delta J^{\mu}$ and $\delta V^{\nu}$. In these I fit ...
2
votes
1answer
37 views

Why are stochastic optimization algorithms able to find global minima (PSO and Genetic algorithm)

Why do these two methods, the particle swarm optimization (PSO) and the genetic algorithm find global minima (or are at least able to). And my second question is that these both algorithms are based ...
2
votes
0answers
41 views

How to find a cost function with only a statistical measure of success?

Using the U.S.A. as a loose analogy, we have search algorithms that find the names and number of States adjacent to a given State (containing a selected city). The goal is to minimize the number of ...