In statistics this refers to selecting an estimator of a parameter by maximizing or minimizing some function of the data. One very common example is choosing an estimator which maximizes the joint density (or mass function) of the observed data referred to as Maximum Likelihood Estimation (MLE).

learn more… | top users | synonyms (2)

0
votes
0answers
10 views

Discrete optimization with a very large solution neighborhood to explore

I have a problem whose feasible (discrete) solutions can measured by a cost function. I am thinking of using some optimization technique to get better solutions from a rough initial approximation. I ...
1
vote
0answers
15 views

How to visualize a set of many optimizations of posterior simulations of an objective function?

I started by fitting a model: $y = f(X) + \epsilon$. The model includes random effects and coefficients -- there is a lot of heterogeneity in the population (and the data is longitudinal). I then ...
0
votes
0answers
9 views

Multiplicative gradient descent?

The normal gradient descent is additive: $w_{t+1}=w_t-\lambda_t\nabla f(w_t)$, but is there a multiplicative gradient descent that looks something like $w_{t+1}=w_t[-\lambda_t\nabla f(w_t)]$? I know ...
0
votes
0answers
31 views

Is there any optimization algorithms for these problems?

I have modeling the problem with the following equation: $$ \min_{X} L(X)=f(X)-\alpha g(X) + \beta k(X) $$ where $\alpha \gt 0, \beta \gt 0$, and, in my cases, I found the $f(X)-\alpha g(X) \lt 0$ and ...
0
votes
0answers
18 views

Reconstruct a vector with a known vector and residual

I observe $\vec y$ and know $\vec x$. I assume that $\vec y$ mostly consists of $\vec x$, with some added residual $\vec r$. This gives me the problem $\vec y = a\vec x + \vec r$, where $a \in [0, ...
1
vote
0answers
26 views

Lagrange Multipliers in practice

Say we want to minimize the function $f^2({\bf{x}})$, under the constraint $g({\bf{x}})=0$. The classic solution (Method I) is to introduce a Lagrange Multiplier, and solve: $$\frac{\partial ...
2
votes
2answers
50 views

Optimal bin width for two dimensional histogram

There are lots of rules for selecting an optimal bin width in a 1D histogram (see for example) I'm looking for a rule that applies the selection of optimal equal-bin widths on two-dimensional ...
1
vote
0answers
30 views

Optimization in R vs Python, constrained, unconstrained and automatic differentiation?

I am an economics/stat guy who uses quite a bit of optimization (maximum likelihood, simulated maximum likelihood), constrained optimization (mathematical programming w/ equilibrium conditions), ...
0
votes
0answers
19 views

What is the best Global Optimization algorithm to deal with integer values (Binary)

I want to use a global optimization algorithm to find the optimum set of values for n variables (n>100) and all variables take binary values (0 or 1). I am thinking in Genetic Algorithm, Particle ...
3
votes
2answers
122 views

Under what circumstances is the log likelihood function of a point process concave?

I am trying to understand under what circumstances the log likelihood function of a point process concave. Assume that the process can be defined by a conditional intensity function and that the log ...
0
votes
0answers
7 views

Definition of the deterministic annealing method

iI have ran into a shape matching problem and one term which I read about is deterministic annealing. I learnt that it would help to convert discrete problems, e.g. ...
0
votes
0answers
9 views

Feature selection based on cost function

Suppose that we are searching for best features using an optimization algorithm for a classification model (MLP,SNM,Regression,etc...). We should set a cost ...
1
vote
0answers
26 views

Optimize number of layers and neurons with an optimization algorithm

I have a neural network that i want optimize number of hidden layers and neurons in every layer using an optimization algorithm like ...
0
votes
0answers
31 views

How to derive this problem with soft-thresholding method?

The problem is defined as $$ \min_{x} \Bigg\{ a{\|x\|}^2+\frac{b}{2}{\|x-c\|}^2 \Bigg\} $$ where $x\in R^{n \times 1}, c \in R^{n \times 1}$ and $a,b$ are scalars. Equations 2.5 to 2.8 of this paper ...
2
votes
0answers
40 views

Finding the most “uniform” or “least concentrated” density function, subject to moment constraints

Background I want to find a probability measure for a continuous random variable, subject to moment constraints, that is maximally "uniform", as defined below: Definition: Maximally Uniform ...
2
votes
1answer
54 views

Linear Regression with $L_2$: Different penalty strengths yield the same parameters?

Suppose I have model 1: $$Y=aX$$ where $X$ is $n\times1$, consisting of a single feature. suppose I fit this model with $L_2$ penalty with coefficient $\lambda=1$ : $$ \underset{a}{\min} ...
1
vote
1answer
19 views

Holt-Winters and Abnormal termination in LNSRCH

I try to fit data with Holt-Winters function in R. Nevertheless, i am getting the following message: ...
4
votes
2answers
61 views

Do optimization techniques map to sampling techniques?

From any generic sampling algorithm, one can derive an optimization algorithm. Indeed, to maximize an arbitrary function $f: \textbf{x} \rightarrow f(\textbf{x})$, it suffices to draw samples from $g ...
0
votes
1answer
41 views

Best optimization package for employee scheduling problem? [closed]

I am looking to solve the optimization problem described below. Which optimization software package would be best suited for this, considering the requirements specified below? Requirements: 1) Can ...
0
votes
1answer
50 views

Convert continues number to integer number in optimization algorithms in MATLAB

I'm using a continuous optimization algorithm for optimizing neural network's number of neurons in first and second layers besides feature selection so I used this structure for converting continues ...
2
votes
1answer
45 views

MAP estimate of posterior parameters

I have a setup where the joint posterior is written as: $$ P(w, \lambda, \phi \vert y) = P(\phi) \times P(w \vert \lambda) \times P(\lambda) \times \prod_{i=1}^{N}P(y_i \vert w_i, \phi, \lambda) $$ ...
2
votes
1answer
48 views

finding global minima of a random forest estimator

I have a random forest regression model with 1000 trees, having 16 parameters (using python scikit-learn). The estimator can predict a target value with cross validated r2 score of 0.87 +/- 0.03. I ...
1
vote
1answer
27 views

EM for Mixtures of Bernoulli (M-step)

When applying the M-step for a mixture of Bernoulli distributions, one of the parameters in our maximization is the Bernoulli parameter $\mu_{k}$, where $k$ is the index of the "mixture component", ...
1
vote
1answer
19 views

Optimise selection from a set, with constraints

I would like to use R to solve a problem I have. I don't even know what to call a problem of this kind and I'm finding Googling difficult. My guess is that this kind of problem already has R ...
0
votes
0answers
35 views

Parameter optimization of SVM

Currently I am using SVM to perform some classification task. I use libSVM with Matlab interface. From the practical guide of SVM (Link), we know that there are two parameters need to be tuned, namely ...
3
votes
1answer
39 views

Constrained Maximization and Likelihood Ratio Tests for Nested Linear Models

Suppose $\boldsymbol \beta \in \mathbb{R}^k$ is a vector of coefficients for a generalized linear model with $g \left[ E(Y|X) \right] = X\beta$ for a link function $g$ and I wish to test the composite ...
0
votes
1answer
50 views

What statistical method to correct systematic error in the output of a economic optimization model?

I am working with an economic optimization model which attempts to model the dynamics of a certain commodity market (prices, quantities, production etc.) for different frequencies (monthly, quarterly, ...
0
votes
0answers
14 views

Binary event probability optimization

I have a relatively small sample of binary events (50-100 events) that occurred during a time of day (the success rate is closely related to the time of day). I'm grouping these events into hour ...
0
votes
1answer
33 views

Conceptual question on optimization

What is the intuition and the physical meaning of the mathematical expression in convex optimization? When using optimization algorithms like particle swarm or genetic algorithm, do they have ...
1
vote
1answer
32 views

Finding optimal combination of parameters for clustering

I have a spreadsheet with one object per line. Each column contains values that are parameters of my objects (let's say length, width, height, weight, color). I can classify objects based on color and ...
1
vote
0answers
33 views

When does l1 regularisation give a sparse solution?

I was maximising a likelihood function, which is convex. I know that the system has a K-sparse solution. I wanted to know the conditions (or some sufficient conditions) on the likelihood function ...
0
votes
0answers
10 views

optimisation procedures before training SVM

I'm using the LIBSVM in Java for classification with 200 documents in inputs. I build/train the SVM using the same input training data. My response time for preprocessing of documents (tokenization, ...
0
votes
0answers
21 views

Analysis of full factorial with categorical dependent variable and blocking?

I'm working on a research project for which there is some proprietary information that I can't provide here. However, I will do my best to lay out as much information as I can. In this project we ...
0
votes
0answers
16 views

How to use regression analysis to set an optimal price

I am working on a side project with very small dataset where i am trying to figure out the optimal price i should set for a transaction fee (something like payPal). Currently i am using an arbitrary ...
0
votes
1answer
25 views

Constraint optimisation

Consider the constraint optimization $\text{argmin}_{\beta}(f(\beta)+\lambda g(\beta))$ can someone define $\beta(\lambda)$. That is, what is the relationship between $\lambda$ and $\beta$?.
0
votes
1answer
27 views

Marquardt Loglikelihood Calculation in Eviews

I paper I am trying to replicate used Eviews to estimate their state space model (by maximizing the associated maximum likelihood). They used the BHHH and Marquardt algorithms. My question is given ...
1
vote
0answers
30 views

How to fight underfitting in deep neural net

When I started with ANN I thought I'd have to fight overfitting as the main problem. But in practice I can't even get my NN to pass the 20% error rate barrier. I can't even nearly beat my score on ...
1
vote
1answer
36 views

Finding optimal hyperplane

I have a set of vectors $\{V_i\}$ in $n$-dimensional space. There is a number corresponded to each vector $\alpha_i = f(V_i)$ ($\alpha_i$ can be negative). I want to find a hyperplane which would ...
3
votes
0answers
112 views

Better estimator of expected sum than mean

I am trying to find the optimal estimator for the maximal expected $\Sigma X_i$ where $X_i$ is sampled from an unknown distribution which is chosen to be maximal. To clarify and simplify, there are ...
0
votes
1answer
33 views

How to use SVD for dimensionality reduction to reduce columns specifically?

My original data has many more columns (features) than rows (users). I'm trying to reduce the features of my SVD (I need all of the rows). I found one method of doing so in a book called "Machine ...
0
votes
1answer
27 views

What method to use for cluster identification ?

This question is from a confused novice. I have a data set with where each point is located in a 2-D space defined by two objectives (say, X and Y). I wish to identify a set of points from this space ...
0
votes
1answer
85 views

Drunken cockroach - Trying to meet expected value

Imagine that you have $1000 that you can split however you want. You bet in a cockroach run, but it is not the finish that's interesting. You can bet for the cockroach to go left or right, and you ...
0
votes
0answers
28 views

Standard Errors of Transformed Variables

I am carrying out an MLE where some I use a log transformation on the variance parameters which are being optimized. When I calculate the standard errors (se) the se of the transformed variables is ...
0
votes
0answers
17 views

Error running optim function with STAR from book example

I'm running an example of Smooth Transition AR (STAR) Model from the book "Analysis of financial time series, 3rd edition" by Tsay, in section 4.1.3. The script is as follows: ...
2
votes
1answer
43 views

Geometric interpretation of penalized linear regression II

An older question gives an intuitive explanation of how penalized linear regression works, using two separate contours: one for the least square objective, one for the penalty term (i.e. ...
0
votes
0answers
20 views

Fisher Scoring v/s Coordinate Descent for MLE in R

R base function glm() uses Fishers Scoring for MLE, while the glmnet uses the coordinate descent method to solve the same equation ? Coordinate descent is more time efficient than Fisher Scoring as ...
5
votes
1answer
180 views

Stochastic Programming with MCMC

I have just started learning about MCMC (using PyMC), and it seems to be a hammer that can be used to solve a large class of inference and optimization problems. While I understand that there are ...
0
votes
0answers
25 views

Difference between a stochastic and deterministic optimization problem?

I am reading a lot about classifying optimization problems. But i cannot find a real life example that treads the difference between a stochastic and deterministic optimization problem.
0
votes
0answers
19 views

Combining BHHH and Levenberg Marquardt

I already asked a question related to this here: When is Maximum Likelihood the same as Least Squares I know understand how Levenberg Marquardt (LM) can be applied to the objective function. In ...
0
votes
1answer
36 views

Sum of weights in portfolio theory is not equal to 1

I'm trying to understand basic portfolio theory using R. As far as I understood, the sum of the weights of assets must be equal to 1 . But in this link, that teaches how to compute the efficient ...