In statistics this refers to selecting an estimator of a parameter by maximizing or minimizing some function of the data. One very common example is choosing an estimator which maximizes the joint density (or mass function) of the observed data referred to as Maximum Likelihood Estimation (MLE).

learn more… | top users | synonyms (2)

0
votes
1answer
24 views

Is EM feasible when there is no closed form maximization of the expectation of log likelihood?

In every example I've seen of expectation maximization, the E step concludes with an expression of the expectation of log likelihood ( $Q(\theta | \theta^{(t)})$ ) for which a maximum w.r.t. $\theta$ ...
0
votes
1answer
35 views

Does constrained EM algorithm work with bad initial inputs?

When trying to perform constrained optimization using EM algorithm, does EM work if the initial solution (x0) violates the constraints?
1
vote
1answer
18 views

Pythagorean Theorem to Optimize Multiple Variables?

New to stats, so I'm not sure if this is an already established thing or something I just made up that feels good. I have a list of board games that I'm interested in buying based on their price, ...
1
vote
0answers
33 views

Maximum likelihood estimation, how to derive the hessian

I am reading a paper and trying to understand how the authors estimated the standard errors of a set of parameter estimates $[\delta \ \ \phi \ \ \Sigma]$. Below is the loglikelihood function (sorry I ...
-3
votes
0answers
55 views

What are common ill-conditioned problems/mistakes in linear regression? [closed]

Many problems are ill-conditioned and cannot be reliably solved using double precision computer systems. Here is what I know that could happen in linear model, that the problem is ill-conditioned and ...
0
votes
0answers
21 views

Summation constraint matrix

I'm trying to create an assignment script using the R package Rsymphony, but I can't seem to figure out how to convert the summations into an actual constraint matrix. I tried to figure it out based ...
6
votes
1answer
140 views

How do iterative methods for solving maximizing likelihood problems work?

Does anyone know about the computational iteration processes for maximum likelihood estimation? If these set of equations cannot be solved practically then how does the computer solve them?
0
votes
0answers
19 views

Numerical optimization of Monte-Carlo simulated moments

I hope not to post an off-topic related question here as it is related to programming, however, I am faced to a specific problem which I 1) tackle in a statistically oriented language and 2) which I ...
0
votes
0answers
16 views

Scheduling grid optimization

I am trying to optimize the programming of multiple TV channels for a given week. For each show (a day, a time and a TV show) it is possible to forecast in advance the number of people that will watch ...
0
votes
0answers
9 views

Repeated sampling via stochastic approximation to estimate noise

In stochastic approximation algorithms, one is interested in finding zeroes or extrema of functions which cannot be computed directly and that we can only observe via noisy observations. To accomplish ...
0
votes
0answers
38 views

Step-by-step example of reverse-mode automatic differentiation

Not sure if this question belongs here, but it's closely related to gradient methods in optimization, which seems to be on-topic here. Anyway, feel free to migrate if you think some other community ...
1
vote
1answer
14 views

Alternative Applications of Portfolio Optimization [closed]

What other statistical optimizations in the natural and social sciences require the maximization of the difference between the mean and the variance? In other words have an objective function (...
11
votes
6answers
469 views

Why are symmetric positive definite (SPD) matrices so important?

I know the definition of symmetric positive definite (SPD) matrix, but want to understand more. Why are they so important, intuitively? Here is what I know. What else? For a given data, Co-...
0
votes
0answers
12 views

Inbound/Outbound calls integer programming

For modeling on call center schedule optimization with inbound calls, I can use library(lpSolve) from R to get the solutions by ...
0
votes
0answers
19 views

Why is the finding proximal operator a convex problem?

I have been researching and the proximal operator seems to be defined as: $$\DeclareMathOperator*{\argmin}{argmin}\DeclareMathOperator{\prox}{prox} \prox_f(v)=\argmin_x\left(f(x)+\frac{1}{2}\|x-v\|^...
4
votes
1answer
49 views

Constrained optimization algorithm in linear regression

I am interested in the following constrained parameter estimation in linear regression, $$ \min_\beta\sum_{i=1}^{n}(y_i-x_i\beta)^2 + \lambda \sum_{j}^{p}f(\beta_j) $$ where the model is $y=x\beta+e$, ...
0
votes
0answers
10 views

How to use a vector of ranks to predict actual values?

I am interested in this problem of learning a machine learning model to take a vector of ranks as input and predict their numerical values. Let's say I have a matrix $Y$ with shape $m$ (instances) ...
0
votes
0answers
16 views

root finding in CART

Brief Question: I haven't seen repeated "root finding" in CART content online. When I use it I mean it in the sense of: the interior max or min of continuous functions lives at the root (or zero) of ...
1
vote
1answer
14 views

Binary Stochastic Programming with Independent or Positively Correlated Co-efficients

A manufacturer can select a maximum of $N$ stores to fulfill orders from a total of $M$ stores who are looking for inventory, $N\le M$. The case when $N\geq M$ is trivially solved when all stores ...
4
votes
1answer
56 views

Thompson Sampling

I read on Wikipedia that Thompson sampling consists in playing the action ${\displaystyle a \in {\mathcal {A}}}$ according to the probability that this action maximizes the expected reward. This ...
2
votes
1answer
45 views

Regret Minimization with Hidden Markov Processes

Consider a hidden Markov process with two states $\{0, 1\}$ represented with $Z_t$. The transition matrix is unknown, although we can assume it's strongly diagonal (i.e. slow-switching). At any time, ...
2
votes
1answer
71 views

How do I build a regression model with integer constraints on parameters?

My question is similar to: How do I fit a constrained regression in R so that coefficients total = 1? except that I am interested in a solution to the following constraints on the parameters: All $\...
1
vote
1answer
85 views
+50

Can I hack weighted loss function by creating multiple copies of data

Suppose we want to build a binary classifier with weighted loss, i.e., it penalize different types of errors (false positive and false negative) differently. At the same time, the software we are ...
0
votes
0answers
14 views

alternatives to survival analysis or other methods of tackling

Assume "I would like to take a trip with my bicycle but a component in it has a probability distribution of failing depending on the number of kilometers. Given that I want to come back also, what is ...
7
votes
1answer
73 views

What are the impacts of choosing different loss functions in classification to approximate 0-1 loss

We know that some objective functions are easier to optimize and some are hard. And there are many loss functions that we want to use but hard to use, for example 0-1 loss. So we find some proxy loss ...
5
votes
1answer
93 views

In PCA, do the principal components beyond the first optimize any expression?

Given a covariance matrix $\mathbf\Sigma$, the first principal component $u_1$ is the unit vector that maximizes variance $u_1'\mathbf\Sigma u_1$. Do there exist similar expressions that the first $k$ ...
1
vote
0answers
19 views

Regression or fit with an homographic or a linear fractional relation

This question is motivated by exploratory data analysis. I have a number of variables, related to a chemical reaction. Each variable is the quantity of a chemical species produced in different ...
4
votes
0answers
33 views

Residual standard error difference between optim and glm

I try to reproduce with optim the results from a simple linear regression fitted with glm or even ...
0
votes
0answers
30 views

Optimize unknown time series model

I have the MAPE figures of a time series model with multiple parameters for different lags. I can simulate values for the multiple parameters and can get the MAPE values. The problem is that the ...
1
vote
2answers
80 views

How to constrain regression coefficient of two variables to have opposite sign?

I am running a simple linear regression with a few variables but the meanings of the variables are such that certain pairs of variables should have coefficients with opposite signs. How should I ...
0
votes
0answers
32 views

Optimisation of a noisy function in R

I have an C++ program that takes about 50 input parameters and then simulates a something. It then returns one number. The simulation is CPU intensive. Further, the output is noisy, meaning that even ...
1
vote
0answers
14 views

Can expectation maximization be used to optimize a quadratic function? [duplicate]

My knowledge about Expectation Maximization (EM) is limited, from my understanding, EM is just an algorithm to do optimization. It works well when we have some hidden / latent variables, such as ...
0
votes
0answers
9 views

Advice for choosing constrained optimization algorithm

Given a number of potentially s-shaped log-logistic functions (https://en.wikipedia.org/wiki/Log-logistic_distribution), I want to optimize F(x;a,b) (the vertical axis on the picture). Where x (the ...
3
votes
1answer
53 views

Advantages and disadvantages of EM algorithm vs trust region methods for nonlinear optimization

I have a set of observations X that I believe were generated by a mixture of several probability distributions (specifically, two von mises and one uniform). I'd like to find the maximum likelihood ...
1
vote
0answers
22 views
4
votes
1answer
89 views

Adam: stochastic gradient descent?

I would like to get a better idea of stochastic gradient descent algorithms, especially and most important Adam, since I've expierenced reasonable results with Adam and refuse to use something "just ...
1
vote
1answer
67 views

Grouping Combinatorial Optimization

I have a real world problem for which I need to create an optimization algorithm. I have a set A, and a group of sets, let's say 500 sets. I need to find the best combination of them to better match ...
4
votes
1answer
117 views

How can I experiment with Lagrange multiplier in PCA optimization?

Suppose we want to solve following optimization problem (it is a PCA problem in this post) $$ \underset{\mathbf w}{\text{maximize}}~~ \mathbf w^\top \mathbf{Cw} \\ \text{s.t.}~~~~~~ \mathbf w^\top \...
1
vote
0answers
42 views

Finding bounds for function parameter

I have a function $f$, for the sake of simplicity it may be unidimensional function (but what if it's multi-dimensional?). I am interested in finding some value parameter $\theta$ that maximizes it. I ...
-1
votes
1answer
26 views

Is the EM-algorithm the same thing that variational inference in LDA?

I am new in the probabilistic topic modeling, and I need to understand deeply the LDA process, I understand what want to do the inference process in LDA, and I understand too that there is 2 "types" ...
1
vote
1answer
65 views

Find best parameter values: minimise some while maximizing others (in R)?

I have several variables (A, B, C) which are independent statistical measures. Each of these vary for different values of my function's parameter X. The plots of A,B,C look something like this: I ...
3
votes
2answers
95 views

Constrained optimization in R

My data is categorized by two different parameters (say F having n groups and S having m groups) and I want to get a relationship between the two. For example $F = ${$f_1 , f_2 , f_3$} = {$ 10,10,5$} ...
1
vote
0answers
10 views

Choosing a cluster with low variance and many data points

I have data points that have been grouped using k-means clustering. Some of these clusters may only have one data point, which would give them a variance of zero. But I am more interested in the ...
0
votes
0answers
9 views

How can I cluster data points according to the local minima they belong to?

I'm using the genetic algorithm for hyperparameter optimisation. My loss function is the cross-validated loss, that means I can evaluate my loss function but I don't know how it looks like (the shape)....
0
votes
0answers
8 views

Method for a continuous variable for each sub-level of an ordinal variable?

I have three variables. A dependent variable (y) which is continuous, one dependent variable that is ordinal (x1), and one continuous variable which is bound (between 0 and 1, x2). I would like to ...
0
votes
0answers
32 views

Optimization depending on initial start values

I'm running a maximum likelihood of a logit regression, but the estimated parameters value and the loglikelihood value are depending on the value of the algorithm's start. For example, if my start is ...
1
vote
0answers
26 views

Optimizing a function available only through (monte-carlo) stochastic approximation

I am working on a problem where I want to estimate the maximum of a density that I can, in practice, evaluate (pointwisely) using a Monte-Carlo approach (because of intractable integrals). Obviously, ...
3
votes
1answer
94 views

Convergence of L-BFGS in non-convex settings

Is it true that generally L-BFGS may not converge in non-convex settings even if learning rate is really small? For example here L-BFGS diverges, but there are theoretical guarantees on its local ...
1
vote
2answers
33 views

Calculate best interval between peaks

I have a vector of values with zeros and some rare positive value (corresponding to the peaks in the hist) ...
12
votes
4answers
492 views

What is an intuitive explanation for how PCA turns from a geometric problem (with distances) to a linear algebra problem (with eigenvectors)?

I've read a lot about PCA, including various tutorials and questions (such as this one, this one, and this one). The geometric problem that PCA is trying to optimize is clear to me: PCA tries to ...