In statistics this refers to selecting an estimator of a parameter by maximizing or minimizing some function of the data. One very common example is choosing an estimator which maximizes the joint density (or mass function) of the observed data referred to as Maximum Likelihood Estimation (MLE).

learn more… | top users | synonyms (2)

0
votes
1answer
18 views

Is a constrained optimization problem equalivant to its Lagrangian form?

For the following problem: $\text{min:}\ f(x)\\ s.t. \ g(x)\leq t$ Is the above problem equivalent to the following problem? $\text{min:}\ f(x) + \lambda g(x) \\ s.t. \ \lambda\geq0$ where $t$ and ...
0
votes
0answers
4 views

How to train input feature weights for a path optimization problem?

I am working on a general path optimization problem. As many of you know, once the weights on all nodes are determined, one may solve this problem in many different means. Unfortunately, my raw input ...
1
vote
0answers
16 views

How to maximize (optimize) a response, given only a dataset of responses and features?

Let's say I have an n x p dataset. For each n, I have the response, 'y', and p - 1 features associated with it. What is the best way to determine the values of the features that will maximize 'y'? The ...
0
votes
0answers
31 views

What is PRINCIPAL HESSAIN Direction Model, how and where can I use it?

I'm a M.Tech student going through my academic project-work, here I have asked to develop a optimal design and a best equation to fit the data for a Leaching, Solvent extraction and Electrowinning ...
1
vote
2answers
70 views

Strict convexity of Ridge vs Convexity of LASSO

Is there any intuition why the ridge regression is strictly convex, while the LASSO is only convex? Does it have to do with the "corners" of the L1 regularization?
1
vote
0answers
10 views

Making sense of the big world of gradient methods [migrated]

There are many extensions of gradient descent: stochastic-, Nesterov accelerated-, proximal-, conjugate-, dual-, mirrored-, splitted-, coordinate- gradient descend and more. It also appears that many ...
0
votes
0answers
32 views

Parameter uncertainty after non-linear least squares estimation

I've fit a system of non-linear ODE to some experimental data using Levemberg-Marquardt. After the algorithm converged, I estimated the Hessian matrix of the system using: $H = (J^TJ)$ The ...
0
votes
0answers
26 views

Finding best parameters of SVM in matlab

I’m designing a system (using Matlab) that I can optimize parameters of a support vector machine (SVM) with genetic algorithm, harmony search and another optimization algorithms to find the best ...
1
vote
0answers
25 views

Outlier detection in binary classification

I have a question about outlier detection in my system. I’m designing a system (in Matlab) that optimize both features and parameters of a classification method (like mlp) together with optimization ...
3
votes
2answers
39 views

Combining models for prediction based on residual performance

I have never read or seen someone do this before, so I wanted to pose the question here. Suppose I fit a basic linear model, $\text{price of house} = \beta_0 + \beta_1*\text{taxes} + ...
0
votes
1answer
22 views

On the stopping criterion of coordinate descent method for linear SVM with $\ell_1$-regularization

I am trying to implement the coordinate descent method to solve the dual of linear SVM problem, but blocked at the stopping criterion. The dual of linear SVM problem is: $$\min f(\mathbf{x}) = ...
1
vote
0answers
34 views

How to augment lpsolve R optimization solution to run on a hadoop cluster? [closed]

I asked same question on stack overflow but didn't get quite a lot replies...wondering if this forum is better.. I am using R lpsolve package to optimize my transportation model. My code runs fine ...
7
votes
0answers
83 views

When optimizing a logistic regression model, sometimes more data makes things go *faster*. Any idea why?

I've been toying around with logistic regression with various batch optimization algorithms (conjugate gradient, newton-raphson, and various quasinewton methods). One thing I've noticed is that ...
0
votes
0answers
37 views

ML algorithm to find optimal control parameter

I have a training dataset $(X, y) \rightarrow z$. Where $X$ is an $n$th dimensional real vector, $y$ is an integer number in $\{1, 2, 3\}$, and $z$ is a real number. I am looking for machine learning ...
11
votes
2answers
481 views

Why maximum likelihood and not expected likelihood?

Why is it so common to obtain maximum likelihood estimates of parameters, but you virtually never hear about expected likelihood parameter estimates (i.e., based on the expected value rather than the ...
0
votes
0answers
8 views

Directional Derivative of a function containing an Indicator function - Optimiality condition for quantile regressors

I'm trying to understand a passage in Koenker's Quantile regression book (p.33). It says: (note that y,x, are vectors and w is the direction vector) With the first part of the outcome no problem: I ...
2
votes
0answers
11 views

Preconditioning gradient descent

If one is using gradient descent to optimize over a vector space where each of the components is of a different magnitude, I know we can use a preconditioning matrix $P$ so that the update step ...
0
votes
0answers
21 views

how can i use regression model selection technique and to carryout ANOVA for calculated data [duplicate]

how can i use regression model selection technique and to carryout ANOVA for calculated data I have problem regarding to know , what kind of relationship is excised between parameters to produce a ...
1
vote
0answers
15 views

Regularization in unordered vectors

Let suppose we are given two vectors u, v $\in \mathbb{R}^n$ and we want a function that returns $0$ if the ordering of the elements of both vectors are the same or a positive number otherwise, where ...
0
votes
1answer
41 views

Quadratic (programming) Optimization : Multiply by scalar

I have two - likely simple - questions that are bothering me, both related to quadratic programming: 1). There are two "standard" forms of the objective function I have found, differing by ...
2
votes
1answer
76 views

How to solve the problem, that the scale of variables influence the gradient/optimization

I've the problem that, using something related to Fisher-scoring, the gradient, which is usually the sum over a variable times a value which depends upon the parameter we are looking for, the updates ...
1
vote
0answers
11 views

The relations between sampling and optimization

Assume that we have $n$ training data $x_1, ... x_n$ , generated by a probability model $P(x;\theta)$. We want to estimate the parameters $\theta$ of the model based on the observations. In ...
0
votes
2answers
18 views

Returning the inverse of a matrix in a quadratic program (SVM) in cvx optimization package

I am solving the dual QP of an SVM, and using the RBF kernel. As you know, the objective function is of the form $$f(\alpha) = \alpha^T Q \alpha $$ where $\alpha$ is the optimization variable and $Q$ ...
0
votes
0answers
12 views

Microeconomics maximization problem [migrated]

I would like to discuss this question about microeconomics: Martha National County Club is a golf club in an isolated wealthy community and accepts only females as members. There are 1,000 ...
0
votes
0answers
39 views

three variables chemical data analysis optimization

Could anybody help me by giving a solved step by step example of response surface methodology (RSM) using Doehler design for leaching of metals or for chemical processes? My supervisor has given me ...
5
votes
2answers
159 views

Bayesian inferencing: how iterative parameter updates work?

I have been struggling with this for a while. A typical optimisation problem can be viewed as optimising some cost function which is a combination of a data term and a penalty term which encourages ...
0
votes
0answers
27 views

Is it possible to estimate the convex combination of parameters in the IRLS-Framework?

Suppose I want to estimate the parameter $\mathbb{E}(Y)=\mu \ge 0$ with $\mu = a(\alpha)\mu_1(\gamma_1) + \Big(1-a(\alpha)\Big)\mu_2(\gamma_2)$ where $a(\alpha)\in (0,1)$ Using the usual ...
0
votes
0answers
3 views

Choosing optimal constraints for optimisation when the potential constraints are correlated

I am characterising some detectors. Each detector consists of a series of elements, thinking of a CCD in a camera is a reasonable approxiamtion. For each detector I have a distribution representing ...
3
votes
2answers
83 views

Gradient of Gaussian log-likelihood

I'm trying to find the MAP estimate for a model by gradient descent. My prior is multivariate Gaussian with a known covariance matrix. On a conceptual level, I think I know how to do this, but I was ...
0
votes
0answers
27 views

Efficient Solving of one-dimensional Equation without bounds in R

I need to solve the equation $0 = Z + a_1X^{\lambda_1}+a_2X^{\lambda_2} := f(X)$ for $X$, where $\lambda_1$, $\lambda_2 \in \mathbb{Q}^-$ and $Z, a_1, a_2 \in \mathbb{Q}$. Unfortunately I only have a ...
0
votes
0answers
14 views

Choosing the values of a proper subset of features to maximise regression tree output

Suppose I have a regression tree and feature set $X$. Suppose that the feature set is composed of $X:=\{X_0,X_1,...,X_{100}\}$, where each $X_i \sim N(0,\sigma^2)$. Suppose that ...
5
votes
0answers
88 views

Meaning of a convergence warning in glmer

I am using the glmer function from the lme4 package in R, and I'm using the bobyqa optimizer ...
0
votes
0answers
16 views

Maximizing expectations vs Mode maximization

In many statistical problems, I see the following formulation for maximizing rewards: Assuming that my total reward $R$ is the sum of individual rewards $R$: ...
0
votes
0answers
11 views

Levenberg-Marquardt adjustment

I understand that Levenberg-Marquardt adjustment is about adding $\lambda \times\; I$ to second derivative of the log likelihood function, where $I$ is the identity matrix $$-[l''(\theta | Y) - ...
1
vote
0answers
19 views

Newton Raphson Algorithm: negative semi definiteness [duplicate]

I would like to minimise the function $l(\theta|Y)$. Given the Newton's method below $$\theta^{(t+1)} = \theta^{(t)} - \left[l''(\theta\;|\;Y)\right]^{-1} l'(\theta^{(t)}\; | Y)\quad t = 0,1,...$$ ...
0
votes
1answer
41 views

What methods exist for finding optimal splits to discretize continuous data with respect to a target variable

I'm doing some research into methods for discretizing a continuous variable coupled with a binary target variable to find the optimal split points to maxamise a measure of impurity (gini/entropy). ...
1
vote
0answers
23 views

Regression clustering

I am looking for references about classical methods in regression clustering. My problem is the following: I have a cloud of points that are assumed to have been generated by inverse functions with ...
0
votes
0answers
19 views

Segmenting an interval sensibly

Is there a canonical/recommended approach to or algorithm for splitting up an interval with the intent of minimizing the number of segments while keeping a high accuracy? It is essentially an ...
0
votes
1answer
27 views

data prediction by regression or better ways

I am working on data prediction. Given data of a random variable $X$ and $Y$, find out how to predict $Y$ from $X$. I know how to do it by linear regression, $\hat{Y} = kX + b$. But, here, $X$ is ...
0
votes
0answers
4 views

Fast diagonalization for coordinate descent

IN order to perform efficient coordinate descent, I am using SVD to diagonalize my system $$y=Ax$$ to $$y=USV^Tx\\ U^Ty=y_{new}=S(V^Tx)=Sx_{new}\\ y_{new}=Sx_{new}$$ where $S$ is the diagonal matrix ...
1
vote
0answers
18 views

General truncated Poisson distribution: R routines

I face an applicaton with (a possible) five-truncated Poisson distribution - I count periods at least five days long, thus data of the type (5, 8, 12, 5, 5, 10, ...). Much theory and software is ...
0
votes
0answers
31 views

How to optimize neural words embedding (Colbert & Weston, 2008) vectors?

AFAIK, the model works in a following way: We have $S$ - all the sentences in our unlabeled copra. choose $N$ (~100000) most popular words from our unlabeled data = $PW$ chose the hyperparameters ...
3
votes
1answer
55 views

When is Likelihood Function Positive Semidefinite

This may be a very misinformed question, but I cant figure out why its not true. Here goes: According to Wikipedia and this post, the hessian of a likelihood function equals the information matrix, ...
0
votes
0answers
12 views

A simple optimization problem [duplicate]

I am trying to derive ELM going through the basics , please help me out here : $$f = x^Tx$$ $$g = Ax-b $$ The constraint is $Ax-b = 0$ I calculated $J' = f'+\lambda^T g'$ which is $2x+(\lambda^T ...
6
votes
1answer
143 views

Does log likelihood in GLM have guaranteed convergence to global maxima?

My questions is: are generalized linear models (GLMs) guaranteed to converge to a global maximum, and if so why? Furthermore, what constraints are there one link function to insure convexity? My ...
2
votes
2answers
79 views

Example how maximizing and minimizing a function can be equivalent?

I don't understand how sometimes given an optimization problem, a function could get its optimal solution by minimizing or sometimes just by reformulation it becomes maximizing. Can you please give me ...
0
votes
1answer
44 views

Minimizing the norm of a vector of parameters

I'm reading a paper that defines a function $f_w(x)$ that takes input $x$ and parameters $w$ and a set of constraints. There are also training data. The aim is to find the set of parameters $w$ that ...
0
votes
1answer
44 views

Goldfarb Idnani quadratic solver

I am implementing the Support Vector Regression (SVR) algorithm by means of quadratic programming. In order to do that, I am using an optimization library that contains a quadratic solver based on the ...
2
votes
0answers
53 views

How to investigate properties of the trimmed mean for a Cauchy variable?

Let $X_1,X_2, \dots,X_n$ be a sample from a population with distribution function $F(x-\theta)$, where $F$ is symmetric around $0$. The $\alpha$ trimmed mean $T_n(\alpha)=\dfrac{1}{n-2\lfloor ...
1
vote
0answers
22 views

Transformation to bound a parameter into an unbound parameter?

I want to optimize the parameters that minimize a particular function. These parameters are typically lower and upper bounded (i.e. some can only lie between 0 and 1, some only between 4 and 6, etc.). ...