In statistics this refers to selecting an estimator of a parameter by maximizing or minimizing some function of the data. One very common example is choosing an estimator which maximizes the joint density (or mass function) of the observed data referred to as Maximum Likelihood Estimation (MLE).
1
vote
0answers
4 views
Confusion related to dual problem formulation in sparse inverse covariance matrix estimation
I was reading this paper where they are trying to estimate the inverse covariance matrix of the gaussian. What they are trying to maximize the gaussian log likelihood. The primal problem is maximize ...
-1
votes
0answers
53 views
How to implement this maximum likelihood algorithm?
I have a log-likelihood function
$$l(b) = \sum_{i=1}^n \left( \log(b_0 + b_1 x_i + b_2 x_i^2) \right) - \sum_{i=1}^{n+k} \left( \int_a^{x_i}\exp(b_0 + b_1u + b_2u^2 + b_3 u^3) du \right)$$
For ...
0
votes
1answer
16 views
can EM algorithm be applied to my problem? Input data set is based on a function of parameter
I understand EM algorithm is often used for missing data/mixture problem. But can it be used to optimize a particular type of likelihood based on jointly fitting variables and transformations of those ...
1
vote
1answer
39 views
Transformation of binary data
I have a function that I am trying to optimize using Particle Swarm Optimization.
The objective function gets a binary string. These binary strings are candidate solutions of the subject function. I ...
0
votes
1answer
31 views
Confusion related to minimization of a gaussian likelihood function
I have this confusion related to minimization of gaussian likelihood function. The negative of the log likelihood of gaussian distribution is
$-\log \det(Q) + \text{tr}(SQ) + \lambda||Q||_{1}$ where ...
0
votes
1answer
60 views
Minimize a function with respect to a matrix
I have two sets of vectors, A and B. Vectors from set A live in an m-dimensional space, ...
0
votes
0answers
9 views
Solving exponantial equations for maxent with opensource applications or coding [migrated]
I am working on maximum entropy. And I have a problem solving some equations.
Look at these 2 equations:
((2*e^{-a*2-b*2^2}) +(6*e^{-a*6-b*6^2}) +(10*e^{-a*10-b*10^2}) +(14*e^{-a*14-b*14^2})) / ...
0
votes
0answers
29 views
Is it possible for a learning algorithm to learn weights on a different scale?
I'm doing some simple regression. In my training I initialize my weights randomly. Then it converges to a minimum, but I noticed that depending on the initial weights, the algorithm seems to find the ...
2
votes
0answers
69 views
How to fit a standardized generalized hyperbolic distribution to my data using optim?
I want to fit a standardized generalized hyperbolid distribution to my data using the optim command.
Regarding my post here, I think I have to try it manually, by using the optim command.
My problem ...
0
votes
0answers
27 views
Optimizing a Poisson Process
So, I was standing in line for a Paramore concert for a very long time this past weekend because the boyfriend wanted to be up front during the show. I got to thinking, what would be the optimal time ...
1
vote
1answer
60 views
What do I need to consider when using the Hessian to computer S.E.'s?
I use optim() in R to do a lot of MLE. I've used the approach for a lot of problems, but the one I'm working on right now consists of fitting the parameters of the generalized extreme value ...
3
votes
0answers
36 views
Reducing the dimension of an embedding
Let $O \in \mathbb R^{p\times m}$ be a data matrix of observations.
Suppose we are given a model $\mu : \mathbb R^n \rightarrow \mathbb R^m$ which is able to approximately fit the observations. Fix ...
1
vote
1answer
53 views
Forecasting optimization techniques in fantasy baseball
I am currently trying to build a better forecasting model for my fantasy baseball roster. I currently am using commonly accepted projected season statistics (ZiPS from Fangraphs) to determine the ...
1
vote
1answer
45 views
Standardized generalized hyperbolic distribution
I am interested in the standardized version (mean zero, variance one) of the generalized hyperbolic and the hyperbolic distribution. I want to include this in my analysis and therefore I need the ...
0
votes
0answers
22 views
R package for implementing firefly algorithm? [migrated]
I am interested in implementing firefly algorithm to solve some optimization problems. Since the majority of my work is based on R, I am searching a R package to do it.
Through searching the ...
1
vote
1answer
47 views
Cauchy M estimator of regression in R
Was wondering if anyone knows of an R package to estimate the Cauchy-M estimator of regression (see for example the end of this section, but with simultaneous estimation of the scale parameter as in ...
1
vote
0answers
27 views
Issues with solving large sparse linear equations
I have some issues solving sparse linear equations Ax = b
My matrix A is sparse with dimension of 5 million by 5 million. Actually, it is a combination of two matrices. One is tridiagonal and the ...
0
votes
0answers
13 views
CVX-esque convex optimization in R? [migrated]
I need to solve (many times, for lots of data, alongside a bunch of other things) what I think boils down to a second order cone program. It can be succinctly expressed in CVX something like this:
...
3
votes
1answer
40 views
(Prediction) constrained OLS in R
I have a collection of $n$ datapoints $(y_i,\bf{x}_i)$ in $\mathbb{R}^{p+1}$ and would like to estimate the following model in R:
...
1
vote
1answer
48 views
Optim result highly dependent on starting value
I want to fit a standardized Student's-t distribution. The log-likelihood is given by:
\begin{align*}
log \mathcal{L}(\nu | l_1,...,l_n)=\sum_{i=1}^n \left( log \left( (\pi ...
3
votes
1answer
107 views
Logistic regression algorithm in Ruby
I have been using R to calculate logistic regression with many independent variables for a Ruby on Rails web application. However, I can no longer import data from the database to R using RPostgreSQL. ...
1
vote
0answers
41 views
Efficient numerical methods to estimate ARMA models?
I mean exact likelihood based estimation instead of these LS methods.
There are more general nonlinear optimzation methods, but in terms of performance, are there any specific methods for this type ...
0
votes
0answers
34 views
R Quadratic programming and constrained optimization problem [closed]
I want find a vector p which lies in the positive null space of a matrix S and as well minimize the value of a least squares function with respect to p. Constraints are thus that p>0 and S*p=0
Here is ...
1
vote
0answers
17 views
confusion related to the dual of svm
I have a confusion related to the dual of svm
In the main objective function I have
Now to solve the dual of this objective function, I will minimize with respect to the primal variables first to ...
0
votes
0answers
26 views
Confusion related to L2 and L1 SVM
I have this confusion related to L1 and L2 svm. I was reading this paper
I am attaching the screenshot and the part I didn't understand
The part that I didn't understand how it was derived
I ...
0
votes
0answers
32 views
Constrained optimization of logistic function
I am using R for the following optization problem. The following function needs to be optimized under constraints on b2:
...
0
votes
0answers
25 views
coordinated dual descent method and sequential minimal optimization
Libsvm uses the sequential minimal optimization as its main solver while Liblinear uses coordinated dual descent method. What are the major differences between these two methods? Looks like both of ...
-1
votes
1answer
39 views
What's the convergence rate when solving L1 regularized optimization via coordinate descent with tiny step? [closed]
Wondering if there is an established result for the convergence rate when solving L1 regularized optimization via coordinate descent with tiny step? By "tiny step" I mean the step is always set to a ...
2
votes
0answers
49 views
Estimator bias without a closed form?
Given a regression loss function $l(Z,\beta)=||Y-Z\beta||_2 + \lambda \beta^TD\beta + r(X,Z)$ where $X$ is the predictor matrix, I would like to estimate a $Z$ that minimizes the above loss in a ...
4
votes
1answer
52 views
What optimization problem does least angle regression try to solve?
In Hastie et al's Elements of Statistical Learning, it says
Least angle regression (LAR) ... can be viewed as a kind of “democratic” version of forward stepwise regression (Section 3.3.2). As we ...
4
votes
2answers
81 views
Support Vector machine : a simple question
I think that a formulation of SVM for points x with label y is :
$$
\begin{align}
\arg\min_{\substack{u,w,b}} \frac{1}{2} \cdot |w|^2 + C \cdot \sum_{i} u_i \\
s.t.\ \ y_i\cdot (w \cdot x_i + b) ...
7
votes
2answers
145 views
How to choose the right optimization algorithm?
I need to find the minimum of a function. Reading the docs at http://docs.scipy.org/doc/scipy/reference/optimize.html I see that there are several algorithms that do the same thing, i.e. find the ...
1
vote
1answer
103 views
How can one show a Kmeans solution is unique?
Suppose we are given a distribution P and a constant K. We wish to minimize the kmeans objective w.r.t centers ${C1,..Ck}$:
What constraints on $P$ are known to imply that the optimal solution is ...
1
vote
1answer
112 views
How to determine if the data points are linearly separable from an SVM hyperplane
How to know the data points are linearly separable from an SVM hyperplane?
How to get the optimal classifier during iteration process?
How to calculate the complexity of the SVM model?
1
vote
0answers
38 views
Confusion related to convex optimization of a function in a svm paper
I was reading this paper http://www.ist.temple.edu/~vucetic/documents/wang11kdd.pdf related to adaptive multi-hyperplane machine for non linear classification
In that paper, they have mentioned about ...
0
votes
0answers
26 views
Confusion related to a symbol used in strong convexity [closed]
I was reading this wikipedia article related to strong convexity. It's written a function f(x) is strongly convex if
I just didn't get the sign. How is it different from $\ge$. And it also says ...
0
votes
0answers
20 views
Numerical optimization [closed]
I need to solve the following optimization problem.
\begin{aligned}
\underset{\lambda_0,\lambda_1}{\min} & \sum_{t=1}^T\sum_{n\in\{1,4,8,12,16,20\}} \left(\hat y_t^{(n)} - y_t^{(n)}\right)^2 ...
0
votes
0answers
64 views
Best fit of weighted mean?
The title might be somewhat misleading. I am trying to solve the following optimization problem and do not find any clues. Can any one help?
I have a sequence of $n$ observation $s_i$. Each ...
3
votes
1answer
84 views
I have GBs of Event-Based Data. How do I figure out causation?
I have a lot of event-based data about users of our website. For example, data in the format (verb, timestamp). There's about 10 or so different verbs (call them A, B, C, etc).
I'm interested in ...
1
vote
0answers
22 views
Bandits without exploitation: finding the best items with incomplete information
I'm trying to analyze a general game. This is probably well-known, in which case pointers to relevant literature would suffice (but explanation would not be declined!). If it's not standard, of course ...
2
votes
1answer
214 views
Kmeans on “symmetric” data
A set is said to be fully-symmetric if for every x in it, negating one of its components results in y such that y is in the set as well.
A set is said to be semi-symmetric if for every x in it, ...
1
vote
0answers
23 views
Smoothing objective function
I have a current problem where i have a continuous error real function of n-variables, which is computationally expensive and (possibly) has multiple local minima (optima). I want to somehow smooth ...
4
votes
1answer
95 views
What's the difference between Maximizing Conditional (Log) Likelihood or Joint (Log) Likelihood while estimating parameters of a model?
Consider a response y and data matrix X. Suppose I'm creating a model of the form -
y ~ g(X,$\theta$)
(g() could be any function of X and $\theta$)
Now, for estimating $\theta$ using Maximum ...
0
votes
0answers
30 views
Is it possible to use EM in optimization with dependent parameters
I have a dataset from which I want to derive distributions.
More precisely I want to derive mean ($\mu$) and deviation ($\sigma$ and $\sigma_1$ later) for each record. In order to do that I can use ...
0
votes
2answers
68 views
Effective way to speed-up my mass-univariate problem (i.e. a large set of per-point optimization problems)
I am new to numerical methods, and I have to solve a problem of medical imaging. My background is computer science. I have a naive, general question.
Problem Statement:
I have an extremly large set ...
3
votes
2answers
162 views
Minimize the standard deviation of the total values of groups of items (optimization?) in r
I am a total novice, but I think what I'm looking for is help with an optimization problem.
I have a set of about 26 items, each with a certain value. I want to distribute these items into 8 groups ...
3
votes
0answers
85 views
Estimating parameters using Kullback-Leibler or Kolmogorv-Smirnoff via Nelder Mead
I want to find the parameters of a model which specifies a set of classification probabilities, for say M classes. (I'll use the parameters in another model later.)
Given a set of parameters ...
0
votes
1answer
74 views
Global optimization of deep learning models
Apparently, deep neural networks have been making an impact recently. The layer-by-layer training of these networks has made it feasible to construct complex, deep, and well-performing neural ...
0
votes
1answer
84 views
How do we usually select the best combination of parameters of a machine learning model (for a given dataset)?
Am I wrong, or the standard way of optimizing a machine learning model is by evaluating the algorithm over the (initial) dataset for all possible combinations of parameters, and then pick up the one ...
1
vote
1answer
32 views
Learning to predict maximum of parameterized function class
I am interested in a multi-task regression problem: I have a parametrized function $f_x : \mathcal{R}^n -> \mathcal{R}$ where $x \in \mathcal{R}$ is a real-valued parameter. For some values of $x$, ...
