In statistics this refers to selecting an estimator of a parameter by maximizing or minimizing some function of the data. One very common example is choosing an estimator which maximizes the joint density (or mass function) of the observed data referred to as Maximum Likelihood Estimation (MLE).

learn more… | top users | synonyms (2)

0
votes
0answers
4 views

Maximum Likelihood Estimation on Zero-Inflated data using Constraints

I have written a function that evaluates the log-likelihood of Zero-inflated Beta Binomial data: ...
1
vote
0answers
16 views

Determining the objective function for a non-linear minimization problem

I have observed a vector of quantities $\vec y$. I wish to use these to constrain a vector of initial conditions $\vec x$ that are related to $\vec y$ through a non-linear (numerically evaluated) ...
0
votes
0answers
17 views

Local maximization of cross-entropy method

The 'cross-entropy' method of derivative-free reinforcement learning is defined as follows: I've come across two problems here: Namely, proving that CEM does not always reach a local maximum, and ...
0
votes
1answer
11 views

Fitting and comparing distributions based on diverse summary statistics

I have a bunch of samples, about 35, drawn from a fat-tailed distribution. I think it is reasonable to assume that the samples are all drawn from distributions from the same distributional family, ...
0
votes
1answer
17 views

Cross entropy-equivalent loss suitable for real-valued labels

I am building a model whose outputs are between 0-1 and the goal is to minimize a cost function over the predicted values and labels. So far everything seems to be easy but my labels are real-valued ...
-1
votes
0answers
5 views

What's the role of line 30 in the implementation of RMSprop? [on hold]

I'm trying to understand the RMSprop optimization method but I haven't been able to figure out why line 30 is necessary in this implementation of the RMSprop. Could someone please explain it to me? ...
0
votes
0answers
21 views

Optimitzation over multiple curves

I have a set of hundreds of curves similar of the ones in this figure: The graph represents the Contrast (CR) vs. the Background-Variability (BV) of some feature in an image. All points are the ...
3
votes
3answers
103 views

Is there a formula for an s-shaped curve with domain and range [0,1]

Basically I want to convert similarity measures into weights which are used as predictors. The similarities will be on [0,1], and I will restrict the weights to also be on [0,1]. I'd like a ...
6
votes
2answers
138 views

Restricted Maximum Likelihood (REML) Estimate of Variance Component

Let, $$\mathbf y_i = \mathbf X_i\mathbf\beta + \mathbf Z_i\mathbf b_i+ \mathbf\epsilon_i,$$ where $\mathbf y_i\sim N(\mathbf X_i\mathbf\beta, \Sigma_i=\sigma^2\mathbf I_{n_i}+\mathbf Z_i \mathbf ...
2
votes
0answers
34 views

SVM optimality criterion in Bottou, Lin (2006)

My question relates to an alternative optimality criterion for an SVM dual solution derived in Bottou, Lin (2006) in pages 8 and 9. Let: $\alpha^* = (\alpha_1^*,\dots,\alpha_n^*)$ be a dual ...
1
vote
2answers
137 views

Derivation of Support Vector Machine

I actually understood the derivation behind support Vector Machine but I have a doubt about constraint equation. Why we have a constraint equation $\geq1$ if $y_i=1$ and $\leq-1$ if $y_i=-1$? Can ...
0
votes
1answer
28 views

Improvements of Random Search for Hyperparameter Optimization [closed]

Random search is one possibility for hyperparameter optimization in machine learning. I have applied random search to search for the best hyperparameters of a SVM classifier with a RBF kernel. ...
0
votes
0answers
7 views

Adaptive classification model

I have come accross a tipical situation where right now i am aware of very few classes and more classes are likely to come with time. I have training data for theae known classes. Once new classes ...
0
votes
2answers
33 views

Maximize slow function [duplicate]

So from experience (don't judge me, it was years ago...) I know that when a person with little knowledge about a field try to explain a problem to someone with a lot of experience it can easily lead ...
2
votes
0answers
22 views

a question on Edgeworth Expansion

I'm working Edgeworth Expansion. I couldn't understand one thing . Can you help me about that please. $$Z= \frac{\sqrt {n} (\bar {x} -\mu)}{\sigma}$$ converges in distribution to N(0,1) I have ...
0
votes
1answer
57 views

Why is optimisation solved with gradient descent rather than with an analytical solution? [duplicate]

I'm trying to understand why, when trying to minimise an objective function, gradient descent is often used, rather than setting the gradient of the error to zero, and solving it analytically. In ...
0
votes
0answers
9 views

single value to describe separation between 3 or more classes

I want to optimize a pre-treatment process. The optimization should be driven by a cost function that describes how well several groups (between 3 to 6 in fact) are separated, after the pre-treatment. ...
1
vote
0answers
55 views

How to improve a bad long-term forecasting of time series in common case

I have two time series $d_t(t)$, $d_c(t)$, where I'm modelling charge as a function of time. Lengths of time series, $N$ are equal to $101$ data points. For the $d_t(t)$ (test sample, short-term) the ...
0
votes
0answers
23 views
0
votes
0answers
15 views

optimizing 2 factors - Can you use NSGA-II to optimize this?

I have 3 machines A, B and C. I would like to rank the machines based on which machine maximizes Score1 and Score2. Score1 and Score2 are performance measures that rang from 0-100%. Below is some ...
0
votes
0answers
30 views

How do I optimize a bioinformatics pipeline for novel data sets?

I'm putting the finishing touches on a bioinformatics pipeline for omics data. There are many sequential interlocking parts (e.g. model fitting, regression, classification, clustering, etc). The final ...
1
vote
0answers
33 views

EM versus other methods of optimization

What are some good examples of likelihoods which are easily maximized by EM but not by other methods of optimization (e.g., gradient ascent) and vice versa?
0
votes
0answers
10 views

Neural networks SVRG

Why aren't neural networks trained using SVRG(Stochastic Variance Reduced Gradient)? I've searched over internet and haven't found anyone who does so. Is that due to need to compute gradient twice ...
0
votes
0answers
13 views

Optimization with a part of gradients

I am exploring some gradient-based optimization functions. For my complex objective function, it is easy to calculate gradients for some parameters, but very hard for some other parameters. I am ...
3
votes
1answer
51 views

Use z-scores to determine the best strategy for airlines

Most airlines board passengers starting from the back of the plane and then working their way towards the front (after boarding priority classes and passengers). In an episode of Mythbusters, Adam ...
1
vote
0answers
16 views

What are some motivations for using nonnegative least squares?

I'm having a hard time understanding the reasoning behind it. Imagining the case of a single independent variable, if the correlation between it and the dependent is very negative, a nonlinear least ...
0
votes
0answers
9 views

What constraints to use for weights in a weighted kernel?

I want to use a Support Vector Machine classifier with the following weighted RBF kernel: $K(x,y) = exp(-\gamma \sum\limits_{i=1}^n w_{i}(x_{i} - y_{i})^2)$ There is one weight for each feature. ...
3
votes
2answers
66 views

Optimizing multiple objective functions simultaneously

I apologize in advance - I am new to both stackexchange as well as as a lot of statistics/machine learning. This is a question I feel must have some fundamentally obvious answer, but after a lot of ...
1
vote
0answers
29 views

Maximum likelihood with special constraints in R? [closed]

I want to find maximum likelihood estimation of parameters under special constraints. My density have 3 parameters $(a,b,c)$ with bounds $$-\infty<a<\infty,\; 1<b<\infty,\; ...
0
votes
0answers
20 views

Help needed for understanding the derivation in an optimization problem

Here is an optimization problem I came across in the paper titled "Learning the dependency structure of latent factors": And here is the closed-form solution of it for S: where X is a PxN data ...
0
votes
1answer
31 views

How is the function learned in a Deep Neural Network non-convex?

The function learned by a Deep Neural Network is essentially composition of different functions. For ex. in CNN first function is convolution (linear function), max-pooling (convex function) followed ...
0
votes
0answers
15 views

SVRG implementation in Theano/Lasagne

I am trying to implement SVRG optimization algorithm using Theano/Lasagne. I have done my own update function like from lasagne.updates. What I don't understand ...
1
vote
0answers
27 views

Interpretation of the consistency property of a loss function

I am looking for an interpretation of the consistency property of a loss function used for classification (e.g., the SVM's hinge loss: $V(t)=\max(0,1-t)$). I copy from Wikipedia: Furthermore, it ...
0
votes
1answer
32 views

Non-linear function optimisation using nlminb function in R

I have been getting error messages in my attempt to estimate parameters in a non-linear function using nlminb function. The following is the code: ...
1
vote
0answers
16 views

Constrained regression for binary dependent variable

I would like to discuss the methodology for the following case: I have a data for several patients over several years for 5 factors describing the health of a particular patient. Every factor ...
1
vote
0answers
33 views

Estimating Gamma MLE with left truncated data (using R and maxLik)

I'm trying to find the maximum likelihood estimation of the parameters of a Gamma distributed random variable using maxLik. The following code explain what I did: ...
0
votes
0answers
29 views

Statistical optimization Model for which person should be assigned a lead within each state

I am working on creating an optimization model based on sales on a particular website. The system assigns the leads to different sales people. I want to generate a model doesn't randomly assign leads ...
0
votes
0answers
7 views

Find a maximum value for input data

I have a set of 20x20000 input variables and 1x20000 target values. I want to find a best combination of 5 variables out of 20 on a condition that other 15 variables are constant to get the best ...
0
votes
0answers
15 views

Multinomial Logistic Regression aka Softmax Regression

In optimization point of view of generalized linear modeling, there is a transfer function that maps a linear score to a final target. There is also a loss function that is minimized in training to ...
0
votes
0answers
16 views

Strategy for estimating a more complex hidden markov model (HMM)

I have a HMM in mind where emission probabilities change over time (not dependent on state). For example, suppose I have two states and four possible emissions. If in state 1, emission probabilities ...
4
votes
1answer
26 views

Bayesian Optimization for a Stochastic Target that changes over time

Let's say there is a single slot machine that: costs zero to play can only be played once per day has a payout that is conditionally normal and is a function of the date and time. I want to use ...
0
votes
0answers
12 views

fast way to train a classifier on different but overlapping features

I am training a linear classifier repeatedly on different set of overlapping features. I have a 3D grid of features, each time features from a small sphere from a grid are used to train a classifier, ...
0
votes
0answers
10 views

Some clarifications about Stochastic and Online Gradient descent

Stochastic and Online gradient descent are ofte used as synonimous, the reason is because both use a 1-sample estimate of the gradient in the parameters update law. Infact assume we have a process ...
0
votes
1answer
25 views

How do I calculate the tipping point of over/under odds in Football?

I am trying to understand how game odds work. One scenario I came across was the over/under scores for football (soccer) games in the form of a table like this: ...
0
votes
0answers
18 views

Statistical analysis of the results calculated from a stochastic global optimization

I recently use the Basinhopping algorithm [1], a Monte Carlo Markov Chain variant, to find the global minimum of numerical functions (which are not necessarily smooth). Though the global optimization ...
1
vote
1answer
98 views

Newton's method for regression analysis without second derivative

In regression analysis, instead of gradient descent, Newton's method can be used for minimizing the cost function. However, in Newton's method, we need to calculate second derivative too. For ...
0
votes
0answers
21 views

Genetic algorithm for solving the un-certain dimension problem

Introduction Last week, I have learned basic genetic programming using Python to solve a simple problem. I introduce it here: There is a city need for air quality monitoring network. The ...
4
votes
1answer
44 views

In neural nets, why use gradient methods rather than other metaheuristics?

In training deep and shallow neural networks, why are gradient methods (e.g. gradient descent, Nesterov, Newton-Raphson) commonly used, as opposed to other metaheuristics? By metaheuristics I mean ...
0
votes
4answers
128 views

Why is the gradient the best direction to move in?

When optimizing a convex function, doing an update like: $$w_{t+1} =w_{t}+ c\ \nabla(f(w)) $$ is recommended. Why is moving along $\nabla(f(w))$ the fastest way to move closer to the goal? What's the ...
1
vote
2answers
73 views

Lower-bound for total-variation between two sub-gaussian random variables

Problem setup Let $X$ and $Y$ random variables on the real line with following properties: $X$ and $Y$ are $\sigma^2$ sub-gaussian $E[X] = 0$ and $E[Y] = \Delta$. Question Whats the lower ...