In statistics this refers to selecting an estimator of a parameter by maximizing or minimizing some function of the data. One very common example is choosing an estimator which maximizes the joint density (or mass function) of the observed data referred to as Maximum Likelihood Estimation (MLE).

learn more… | top users | synonyms (2)

1
vote
1answer
9 views

How to find $\arg\max$ of a neural network?

Let's say I have a neural network $f$ that takes input $\vec x \in \mathbb {R}^n$ and produces output $f(\vec x) \in \mathbb{R}$. How can I find $\hat x = \underset{\vec x}{\arg\max} \; f(\vec x)$?
0
votes
0answers
11 views

Online EM?? Did any one implement online EM using matlab?

In Online EM algorithm in each iteration for each sample one estimates sufficient statistics and adds it to overall sufficient statistics and in maximization step one maximizes parameters to MLE for ...
0
votes
0answers
13 views

maximizing matrix diagonal elements in R [migrated]

Given a matrix (or table in R ) ...
4
votes
2answers
49 views

Expectation maxmisation algorithm increases true likelihood at each iteration

I've heard that the EM algorithm ensures that the true likelihood is non-decreasing at each iteration of the algorithm, but I'm not sure why this is the case. I've provided a basic plot which I ...
3
votes
1answer
40 views

What are the differences between various R quadratic programming solvers?

I am looking for a package to help me solve some quadratic optimisation problems and I see there are at least half a dozen different packages. According to this page: QP (Quadratic programming, ...
3
votes
1answer
21 views

What is the solution to this minimization problem?

I'm encountering the following minimization problem in my research: $$\hat b = \underset{b}{\arg\min} \sum_i^n \left( \log \frac{a_i}{b} \right)^2$$ I could iteratively optimize, but I think that ...
1
vote
0answers
14 views

Under what assumptions can parameter estimate uncertainty be estimated from the Hessian?

Given a model with some parameters, some data it's attempting to reproduce, and a distance function to quantify how well the predictions correspond to the data, I can fit parameters via a general ...
0
votes
1answer
12 views

Total Variation Denoising help

I am trying to work through the "Mathematical exposition for 1D digital signals" in the wikipedia entry for Total Variation Denoising (TVD). I am familiar with Lagrange multipliers. However, I cant ...
1
vote
0answers
14 views

Likelihood convexification

I am doing constrained vector optimization using a non-convex non-linear likelihood function. My problem is of the following form: $$\begin{align*}\hat Q &= \underset{\vec Q}{\arg\min} -\log ...
0
votes
0answers
20 views

Implementing evolutionary algorithms

We're trying to to minimize the following functions using Multi Objective Evolutionary Algorithms : $\mathrm{minimize}~\lambda_q(1-a)E(N)E(X)+\lambda_xE(N)E(\max(X/a-R,0))$ ...
0
votes
1answer
28 views

Clarification about Perceptron Rule vs. Gradient Descent vs. Stochastic Gradient Descent implementation

I experimented a little bit with different Perceptron implementations and want to make sure if I understand the "iterations" correctly. Rosenblatt's original perceptron rule As far as I understand, ...
0
votes
0answers
15 views

What is Tau and Omega in the Black Litterman model?

I'm looking into the BL model when it comes to portfolio optimization, and I'm having a hard time trying to understand each one. I've read on several papers that Omega is the covariance matrix, but I ...
0
votes
2answers
41 views

R optim function - Setting constraints for individual parameters

I am looking to minimize a function using optim as follows: ...
0
votes
1answer
51 views

R - Constrained optimization for a function taking a matrix

I would like to do constrained optimization for a function which takes a matrix as input. The matrix is sparse, representing a weighted adjacency matrix , and only the weights shall be subject to ...
0
votes
0answers
9 views

Prove a property of primal-dual problems [migrated]

When I was studying the computation aspects of quantile regression, I consulted some linear programming book and found an interesting property as follows: If the primal problem have unbounded ...
2
votes
1answer
90 views

Constrained assignment problem (Linear Programming, Genetic Algorithm, etc…)

I'm looking for advice on how I should approach a specific problem. I have about 1000 shops that I have to assign to about 20 different supply centers out of a possible 28, and I'm trying to pick the ...
1
vote
1answer
21 views

On the usage of numerical optimization technique to maximize log-likelihood

Literature and resources say that when the ML log-likelhood does not have a closed form expression, then we can use Newton-Raphson and other optimization techniques. My Question is: During estimation ...
-1
votes
0answers
12 views

Constrained max using Rglpk in R, not conformable [migrated]

I have a question regarding constrained maximization using the Rglpk packaged in R. In the code below, we generate a dataframe 'df'. I am trying to max the sum of column z, subject to: The COUNT ...
0
votes
1answer
56 views

Selecting optimization algorithms

I am in the process of developing a multi-state model (using the msm package in R), and have been encountering the error message initial value in 'vmmin' is not finite This error occurs ...
0
votes
0answers
38 views

Choose the correct regression model

The relation of two measurement values x and y might be linear mod1 <- lm(x~y) ...
0
votes
0answers
14 views

SGD weight learning for mixture of generative models

Let's say that I've learned $k$ generative models $\mathbf{G}=G_i$ in some ways from my data. I'd like to create a mixture from them in order to have something like model averaging in this form: ...
0
votes
0answers
15 views

What is the derivative of this? [migrated]

I have a function of the following form: $J = \|W^TW-I\|_F^2$ Where, $W$ is a matrix and $F$ is the Frobenius Norm. How can I find the derivative of $\frac{\partial J}{\partial W}$ ?
3
votes
1answer
63 views

Designing Asymmetric regression (assymettric loss for regression)

I have a hybrid classification/regression problem.The predicted value can be assumed to be centred around 0. I want to penalize the predictor more, if the predicted value and actual value have ...
1
vote
1answer
18 views

Finding all solutions of a multidimensional nonlinear equation

I have a problem in which I have two (though there can be more) multivariate Gaussian functions very close to each other and I want to find whether being next together causes the sum of these two ...
1
vote
1answer
76 views

Nonlinear Autoregressive model parameter estimation from time series

I'm working on a nonlinear multivariate autoregressive model of order 1 (markovian). It is a discrete-time dynamical system which models exchange of mass between compartments in a compartmental model ...
0
votes
0answers
25 views

Maximum likelihood method

I try to calibration the parameters $\theta$ of my probabilistic model from available (limited) information at some point $\mathbf{x}_i, i=(1,...,n)$ on a 3D space defined by $[0, M]^3 \subset ...
0
votes
1answer
75 views

What is the connection between regularization and the method of lagrange multipliers ?

To prevent overfitting people people add a regularization term (proportional to the squared sum of the parameters of the model) with a regularization parameter $\lambda$ to the cost function of linear ...
0
votes
0answers
13 views

Pairwise elastic net Algorithms

I am interested in minimizng the following $f(x)=x^{T}Rx + \sum_{j=1}^{N}\sum_{i=1}^{N}A_{i,j}|x_{i}||x_{j}| +\sum_{i=1}^{N} |x_{i}|$ where the $R,A$ are positive definite and $A_{i,j}>0$ I ...
0
votes
0answers
7 views

confidence intervals in multivariate nonlinear model [duplicate]

I am recently start to use this site which is very helpful for learning purpose.I am basically engineer and facing the problem that is related to construction of confidence intervals .Mathematically ...
0
votes
0answers
24 views

Marketing offer optimization

I would like to set up a campaign for 2 products (2 offers each) Product A - offer 1 - 25% off Product A - offer 2 - 50% off Product B - offer 3 - 25% off Product B - offer 4 - 50% off Data ...
5
votes
1answer
218 views

Default lme4 optimizer requires lots of iterations for high-dimensional data

TL;DR: lme4 optimization appears to be linear in the number of model parameters by default, and is way slower than an equivalent ...
2
votes
0answers
108 views

A smarter way for evaluating combinations of samples to optimize an overall score

I have the following problem (which I simplified for clarity) that I want to find k out of n samples (here: rows) that yield the maximum score (sum of the values in ...
2
votes
1answer
60 views

how to differentiate matrix

How can I differentiate the following by $\mathbf{W}$ ? \begin{equation} \mathbf{Y} = (\mathbf{W}^T\mathbf{x} + b)^2 \end{equation} Where $\mathbf{W} \in \mathcal{R}^{d\times D}$ and ...
3
votes
1answer
48 views

maximum a posteriori vs squared loss

I am unclear about max a posteriori and squared loss. Let me assume I have $N$ images and $\mathbf{y}_i$ is the label of the image $i$, where, $\mathbf{y}_i\in \mathbb{R}^{C\times 1}$ - a binary ...
4
votes
2answers
144 views

Statistical analysis in the cryptoanalysis of the Enigma cipher machine (safeguarding of the intelligence source in Ultra)

During the second world war the British were doing cryptanalysis of the German Enigma cipher machine and subsequent signals intelligence. One of the issue was about the safeguarding of the source of ...
1
vote
1answer
56 views

optimal down payment estimation in credit scoring

Knowing I can estimate the risk of default, via logistic regression, of a consumer on a small loan... what would be the best way to estimate the optimal down-payment amount to ask for in order to ...
0
votes
0answers
29 views

Optimal number of workers for small business model

I'm new here. I formed an analytical model of a small scale business where the expense can be defined as $C_L=(1-T).(1-1/n)$ and production rate can be defined as $R=1/(1-T+T/n)$. Where ...
2
votes
0answers
89 views

Minimizing KL divergence from a given distribution, according to a graph

Given $n$ discrete random variables $X_1,...,X_n$, a distribution $p$ on $X=(X_1,...,X_d)$ and a DAG (Directed Acyclic Graph) $G$ on $\{1,...,d\}$, which is the distribution $q$ factorizing with $G$ ...
0
votes
1answer
14 views

Compare convergence of optimization methods

I need to quantify how 2 optimization methods differ in convergence. When training a neural network I get the following plots, which show an error function after each gradient update. I think the ...
2
votes
1answer
100 views

How do CNN's avoid the vanishing gradient problem

I have been reading alot about convoloutional neural networks and was wondering how they avoid the vanishing gradient problem. I know deep belief networks stack single level auto-encoders or other ...
1
vote
0answers
23 views

Time series comparisons: early detection of mismatching series after n points for efficiency

I am doing time series comparisons. I have a set of values (my query set Q) that I need to compare against many other reference sets (R), each of which contains the same number of values as my query ...
0
votes
1answer
42 views

Update Rules in Expectation Maximization

I am emulating a certain PDF behaviour using a function. However, due to divergent improper integral, I don't have a closed form expression for the normalization constant. To get the PDF, I just ...
0
votes
0answers
37 views

A constrained Dynamic programming problem

I'm working on a linear time varying discrete(LTV) multi input multi output(mimo) system. I formulate the problem description in the following way $$x_i(k+1) = x_i(k)\cdot A_i(k) + B_i(k)\cdot ...
1
vote
0answers
50 views

Equivalence of the partial least square regresssion's iterative algorithm and its optimization problem

I am reading The Elements of Statistical Learning. This is a page from the partial least square section: The exercise asks to prove the equivalence between Algorithm 3.3 and Eq. (3.64). Here's my ...
4
votes
3answers
109 views

Decompose a time series into superposition of step functions?

Background I have time series data comprising hourly observations of a sensor's readings over a period of almost a year. The sensor records an environment whose baseline measurements should have ...
1
vote
0answers
12 views

Solving for primary variables of a linear program after already having solved for the dual

I was wondering if there is a general procedure of solving for the primary variables of a linear or quadratic or, in general, a convex program after already having solved the dual program. The ...
1
vote
0answers
15 views

Effect of dimensionality on gradient descent

How does increase in number of dimensions effect the convergence of gradient descent method. How is number of saddle points, local extrema related to dimensionality of data?
1
vote
0answers
38 views

The optimization problem of soft margin Support Vector Machine: How to interpret?

I try to understand what exactly we are trying to optimize in the case of Support Vector Machine problem, which supports soft margins. The original problem is posed first as, without soft margins ...
0
votes
0answers
38 views

Polynomial fitting for Shading Correction

I am reading the book Optimization for Computer Vision, and the first example of optimization is a regression for shading correction, in which the author proposes the following polynomial: $p_s(c,x) ...
1
vote
1answer
25 views

Is this problem convex ? (regularization term on xTw)

Suppose we want to solve the following: $$ \min_{w} f(x^Tw, y) + \lambda g(x^Tw) $$ with $f$ a (logistic) loss and $g$ something like a variance. Is this a convex optimization problem ? What are ...