In statistics this refers to selecting an estimator of a parameter by maximizing or minimizing some function of the data. One very common example is choosing an estimator which maximizes the joint density (or mass function) of the observed data referred to as Maximum Likelihood Estimation (MLE).

learn more… | top users | synonyms (2)

0
votes
0answers
12 views

Principal Component Analysis and generalized variance

I'm trying to prove that the first $k$ directions found by PCA maximize the generalized variance, i.e. the determinant of the covariance matrix. Basically, I'm trying to prove that $$ \...
0
votes
0answers
37 views

Recommend a optimization book with more coding examples?

I am interested in continuous optimization problems. However, I feel it is very difficult for me to understand the classic books such as Convex Optimization or Numerical Optmization. My problem with ...
3
votes
1answer
49 views

Adam: stochastic gradient descent?

I would like to get a better idea of stochastic gradient descent algorithms, especially and most important Adam, since I've expierenced reasonable results with Adam and refuse to use something "just ...
0
votes
0answers
34 views

Grouping Combinatorial Optimization

I a have a real world problem for which I need to create an optimization algorithm. I have a set A, and a group of Sets, lets say 500 sets. I need to find the best combination of them to better ...
2
votes
1answer
63 views

How can I experiment with lagrange multiplier in PCA optimization

Suppose we want to solve following optimization problem (it is a PCA problem in this post) $$ \underset{\mathbf w}{\text{maximize}}~~ \mathbf w^\top \mathbf{Cw} \\ \text{s.t.}~~~~~~ \|\mathbf w\|_2=1 ...
1
vote
0answers
38 views

Finding bounds for function parameter

I have a function $f$, for the sake of simplicity it may be unidimensional function (but what if it's multi-dimensional?). I am interested in finding some value parameter $\theta$ that maximizes it. I ...
0
votes
1answer
19 views

Is the EM-algorithm the same thing that variational inference in LDA?

I am new in the probabilistic topic modeling, and I need to understand deeply the LDA process, I understand what want to do the inference process in LDA, and I understand too that there is 2 "types" ...
0
votes
1answer
60 views

Find best parameter values: minimise some while maximizing others (in R)?

I have several variables (A, B, C) which are independent statistical measures. Each of these vary for different values of my function's parameter X. The plots of A,B,C look something like this: I ...
3
votes
2answers
92 views

Constrained optimization in R

My data is categorized by two different parameters (say F having n groups and S having m groups) and I want to get a relationship between the two. For example $F = ${$f_1 , f_2 , f_3$} = {$ 10,10,5$} ...
0
votes
0answers
8 views

Choosing a cluster with low variance and many data points

I have data points that have been grouped using k-means clustering. Some of these clusters may only have one data point, which would give them a variance of zero. But I am more interested in the ...
0
votes
0answers
7 views

How can I cluster data points according to the local minima they belong to?

I'm using the genetic algorithm for hyperparameter optimisation. My loss function is the cross-validated loss, that means I can evaluate my loss function but I don't know how it looks like (the shape)....
0
votes
0answers
7 views

Method for a continuous variable for each sub-level of an ordinal variable?

I have three variables. A dependent variable (y) which is continuous, one dependent variable that is ordinal (x1), and one continuous variable which is bound (between 0 and 1, x2). I would like to ...
0
votes
0answers
29 views

Optimization depending on initial start values

I'm running a maximum likelihood of a logit regression, but the estimated parameters value and the loglikelihood value are depending on the value of the algorithm's start. For example, if my start is ...
1
vote
0answers
24 views

Optimizing a function available only through (monte-carlo) stochastic approximation

I am working on a problem where I want to estimate the maximum of a density that I can, in practice, evaluate (pointwisely) using a Monte-Carlo approach (because of intractable integrals). Obviously, ...
2
votes
1answer
73 views

Convergence of L-BFGS in non-convex settings

Is it true that generally L-BFGS may not converge in non-convex settings even if learning rate is really small? For example here L-BFGS diverges, but there are theoretical guarantees on its local ...
0
votes
0answers
8 views

Finding Hot Spots in Python. KernelDensity [migrated]

I am facing the following problem: I have (large) sample of unevenly distributed points $(X_i,Y_i)$ in a 2D space. I would like to determine the local extremas of the density of the distribution. ...
1
vote
2answers
31 views

Calculate best interval between peaks

I have a vector of values with zeros and some rare positive value (corresponding to the peaks in the hist) ...
9
votes
4answers
379 views

What is an intuitive explanation for how PCA turns from a geometric problem (with distances) to a linear algebra problem (with eigenvectors)?

I've read a lot about PCA, including various tutorials and questions (such as this one, this one, and this one). The geometric problem that PCA is trying to optimize is clear to me: PCA tries to ...
1
vote
0answers
11 views

observed external cost function

Imagine data like this: 100 input nodes 5 output nodes Cost function: a scalar. So yeah, this is an optimization problem, not a "model". What I want to find is: given set of inputs what are the ...
0
votes
0answers
6 views

Selecting input variable values for optimized outputs

I have a question where I have some (n) inputs and one output. I have about 15000 training examples of such data. My problem is as follows: In the input, some (m) variables can be selected by me, ...
0
votes
0answers
18 views

How to propagate “model covariance” into a covariance matrix?

I have a theory $f$ (actually a set of coupled non-linear differential equations) that, from a vector of $n$ initial conditions $\vec x$, is able to predict $m$ values $f(\vec x) = \vec y$. I can ...
22
votes
1answer
611 views

What are the classical notations in statistics, linear algebra and machine learning? And what are the connections between these notations?

When we read a book, understanding the notations plays a very important role of understanding the contents. Unfortunately, different communities have different notation conventions for the formulation ...
2
votes
1answer
20 views

Extension to SAFE screening rule for Lasso

In El Ghaoui et al. (2010), "Safe feature elimination in sparse learning" and following works, screening rules are derived for Lasso (as well as other L1-penalized problems): $ \min_w \|y-X w\|^2 + \...
0
votes
0answers
24 views

Using neural networks to select inputs or input ranges for optimizing output?

I have been given a project to apply computational intelligence to optimize output of a process. I have been given some 150 parameters, which have several constraints on their numerical values. The ...
1
vote
1answer
46 views

R - Constrained Optimization of a Function with Large Matrix Input Having Both Fixed and Dynamic Variables

I am trying to run an optimization on a ROI function with a large matrix (~9,000 rows and 15 columns) as an input. A sample of my data structure is below: ...
0
votes
0answers
15 views

Maximum Likelihood Estimation on Zero-Inflated data using Constraint

I have written a function that evaluates the log-likelihood of Zero-inflated Beta Binomial data: ...
1
vote
0answers
22 views

Determining the objective function for a non-linear minimization problem

I have observed a vector of quantities $\vec y$. I wish to use these to constrain a vector of initial conditions $\vec x$ that are related to $\vec y$ through a non-linear (numerically evaluated) ...
0
votes
0answers
37 views

Local maximization of cross-entropy method

The 'cross-entropy' method of derivative-free reinforcement learning is defined as follows: I've come across two problems here: Namely, proving that CEM does not always reach a local maximum, and ...
0
votes
1answer
16 views

Fitting and comparing distributions based on diverse summary statistics

I have a bunch of samples, about 35, drawn from a fat-tailed distribution. I think it is reasonable to assume that the samples are all drawn from distributions from the same distributional family, ...
0
votes
1answer
33 views

Cross entropy-equivalent loss suitable for real-valued labels

I am building a model whose outputs are between 0-1 and the goal is to minimize a cost function over the predicted values and labels. So far everything seems to be easy but my labels are real-valued ...
0
votes
0answers
23 views

Optimitzation over multiple curves

I have a set of hundreds of curves similar of the ones in this figure: The graph represents the Contrast (CR) vs. the Background-Variability (BV) of some feature in an image. All points are the ...
3
votes
3answers
111 views

Is there a formula for an s-shaped curve with domain and range [0,1]

Basically I want to convert similarity measures into weights which are used as predictors. The similarities will be on [0,1], and I will restrict the weights to also be on [0,1]. I'd like a ...
6
votes
2answers
152 views

Restricted Maximum Likelihood (REML) Estimate of Variance Component

Let, $$\mathbf y_i = \mathbf X_i\mathbf\beta + \mathbf Z_i\mathbf b_i+ \mathbf\epsilon_i,$$ where $\mathbf y_i\sim N(\mathbf X_i\mathbf\beta, \Sigma_i=\sigma^2\mathbf I_{n_i}+\mathbf Z_i \mathbf G\...
2
votes
0answers
47 views

SVM optimality criterion in Bottou, Lin (2006)

My question relates to an alternative optimality criterion for an SVM dual solution derived in Bottou, Lin (2006) in pages 8 and 9. Let: $\alpha^* = (\alpha_1^*,\dots,\alpha_n^*)$ be a dual ...
1
vote
2answers
145 views

Derivation of Support Vector Machine

I actually understood the derivation behind support Vector Machine but I have a doubt about constraint equation. Why we have a constraint equation $\geq1$ if $y_i=1$ and $\leq-1$ if $y_i=-1$? Can ...
0
votes
1answer
30 views

Improvements of Random Search for Hyperparameter Optimization [closed]

Random search is one possibility for hyperparameter optimization in machine learning. I have applied random search to search for the best hyperparameters of a SVM classifier with a RBF kernel. ...
0
votes
0answers
7 views

Adaptive classification model

I have come accross a tipical situation where right now i am aware of very few classes and more classes are likely to come with time. I have training data for theae known classes. Once new classes ...
0
votes
2answers
33 views

Maximize slow function [duplicate]

So from experience (don't judge me, it was years ago...) I know that when a person with little knowledge about a field try to explain a problem to someone with a lot of experience it can easily lead ...
2
votes
0answers
22 views

a question on Edgeworth Expansion

I'm working Edgeworth Expansion. I couldn't understand one thing . Can you help me about that please. $$Z= \frac{\sqrt {n} (\bar {x} -\mu)}{\sigma}$$ converges in distribution to N(0,1) I have ...
0
votes
1answer
69 views

Why is optimisation solved with gradient descent rather than with an analytical solution? [duplicate]

I'm trying to understand why, when trying to minimise an objective function, gradient descent is often used, rather than setting the gradient of the error to zero, and solving it analytically. In ...
0
votes
0answers
10 views

single value to describe separation between 3 or more classes

I want to optimize a pre-treatment process. The optimization should be driven by a cost function that describes how well several groups (between 3 to 6 in fact) are separated, after the pre-treatment. ...
1
vote
0answers
58 views

How to improve a bad long-term forecasting of time series in common case

I have two time series $d_t(t)$, $d_c(t)$, where I'm modelling charge as a function of time. Lengths of time series, $N$ are equal to $101$ data points. For the $d_t(t)$ (test sample, short-term) the ...
0
votes
0answers
25 views
0
votes
0answers
18 views

optimizing 2 factors - Can you use NSGA-II to optimize this?

I have 3 machines A, B and C. I would like to rank the machines based on which machine maximizes Score1 and Score2. Score1 and Score2 are performance measures that rang from 0-100%. Below is some ...
0
votes
0answers
30 views

How do I optimize a bioinformatics pipeline for novel data sets?

I'm putting the finishing touches on a bioinformatics pipeline for omics data. There are many sequential interlocking parts (e.g. model fitting, regression, classification, clustering, etc). The final ...
1
vote
0answers
42 views

EM versus other methods of optimization

What are some good examples of likelihoods which are easily maximized by EM but not by other methods of optimization (e.g., gradient ascent) and vice versa?
0
votes
0answers
15 views

Neural networks SVRG

Why aren't neural networks trained using SVRG(Stochastic Variance Reduced Gradient)? I've searched over internet and haven't found anyone who does so. Is that due to need to compute gradient twice ...
0
votes
0answers
13 views

Optimization with a part of gradients

I am exploring some gradient-based optimization functions. For my complex objective function, it is easy to calculate gradients for some parameters, but very hard for some other parameters. I am ...
3
votes
1answer
53 views

Use z-scores to determine the best strategy for airlines

Most airlines board passengers starting from the back of the plane and then working their way towards the front (after boarding priority classes and passengers). In an episode of Mythbusters, Adam ...