In statistics this refers to selecting an estimator of a parameter by maximizing or minimizing some function of the data. One very common example is choosing an estimator which maximizes the joint density (or mass function) of the observed data referred to as Maximum Likelihood Estimation (MLE).

learn more… | top users | synonyms (2)

0
votes
0answers
15 views

What is the derivative of this? [migrated]

I have a function of the following form: $J = \|W^TW-I\|_F^2$ Where, $W$ is a matrix and $F$ is the Frobenius Norm. How can I find the derivative of $\frac{\partial J}{\partial W}$ ?
1
vote
1answer
18 views

Asymmetric regression (assymettric loss for regression)

I have a hybrid classification/regression problem. The predicted value can be assumed to be centred around 0. I want to penalize the predictor more, if the predicted value and actual value have ...
0
votes
0answers
16 views

Gradient function for restricted likelihood with respect to terms that influence Sigma

Is there a straightforward/generalized way to calculate partial derivatives for the gradient of the restricted multivariate log-likelihood function $\ln\mathscr{L}=C+\ln\lvert ...
1
vote
1answer
15 views

Finding all solutions of a multidimensional nonlinear equation

I have a problem in which I have two (though there can be more) multivariate Gaussian functions very close to each other and I want to find whether being next together causes the sum of these two ...
1
vote
1answer
57 views

Nonlinear Autoregressive model parameter estimation from time series

I'm working on a nonlinear multivariate autoregressive model of order 1 (markovian). It is a discrete-time dynamical system which models exchange of mass between compartments in a compartmental model ...
0
votes
0answers
23 views

Maximum likelihood method

I try to calibration the parameters $\theta$ of my probabilistic model from available (limited) information at some point $\mathbf{x}_i, i=(1,...,n)$ on a 3D space defined by $[0, M]^3 \subset ...
0
votes
1answer
30 views

What is the connection between regularization and the method of lagrange multipliers ?

To prevent overfitting people people add a regularization term (proportional to the squared sum of the parameters of the model) with a regularization parameter $\lambda$ to the cost function of linear ...
0
votes
0answers
10 views

Pairwise elastic net Algorithms

I am interested in minimizng the following $f(x)=x^{T}Rx + \sum_{j=1}^{N}\sum_{i=1}^{N}A_{i,j}|x_{i}||x_{j}| +\sum_{i=1}^{N} |x_{i}|$ where the $R,A$ are positive definite and $A_{i,j}>0$ I ...
0
votes
0answers
8 views

Differences between method=“parRF” and method=“rf” [migrated]

Want to optimize computation time for random forests and the caret package has a built-in train function that allows running parallel random forests. I'm new to the caret package, so don't understand ...
0
votes
0answers
6 views

confidence intervals in multivariate nonlinear model [duplicate]

I am recently start to use this site which is very helpful for learning purpose.I am basically engineer and facing the problem that is related to construction of confidence intervals .Mathematically ...
0
votes
0answers
21 views

Marketing offer optimization

I would like to set up a campaign for 2 products (2 offers each) Product A - offer 1 - 25% off Product A - offer 2 - 50% off Product B - offer 3 - 25% off Product B - offer 4 - 50% off Data ...
5
votes
1answer
156 views

Default lme4 optimizer requires lots of iterations for high-dimensional data

TL;DR: lme4 optimization appears to be linear in the number of model parameters by default, and is way slower than an equivalent ...
2
votes
0answers
106 views

A smarter way for evaluating combinations of samples to optimize an overall score

I have the following problem (which I simplified for clarity) that I want to find k out of n samples (here: rows) that yield the maximum score (sum of the values in ...
2
votes
1answer
56 views

how to differentiate matrix

How can I differentiate the following by $\mathbf{W}$ ? \begin{equation} \mathbf{Y} = (\mathbf{W}^T\mathbf{x} + b)^2 \end{equation} Where $\mathbf{W} \in \mathcal{R}^{d\times D}$ and ...
3
votes
1answer
41 views

maximum a posteriori vs squared loss

I am unclear about max a posteriori and squared loss. Let me assume I have $N$ images and $\mathbf{y}_i$ is the label of the image $i$, where, $\mathbf{y}_i\in \mathbb{R}^{C\times 1}$ - a binary ...
4
votes
2answers
82 views

Statistical analysis in the cryptoanalysis of the Enigma cipher machine (safeguarding of the intelligence source in Ultra)

During the second world war the British were doing cryptanalysis of the German Enigma cipher machine and subsequent signals intelligence. One of the issue was about the safeguarding of the source of ...
1
vote
1answer
49 views

optimal down payment estimation in credit scoring

Knowing I can estimate the risk of default, via logistic regression, of a consumer on a small loan... what would be the best way to estimate the optimal down-payment amount to ask for in order to ...
0
votes
0answers
22 views

Optimal number of workers for small business model

I'm new here. I formed an analytical model of a small scale business where the expense can be defined as $C_L=(1-T).(1-1/n)$ and production rate can be defined as $R=1/(1-T+T/n)$. Where ...
2
votes
0answers
86 views

Minimizing KL divergence from a given distribution, according to a graph

Given $n$ discrete random variables $X_1,...,X_n$, a distribution $p$ on $X=(X_1,...,X_d)$ and a DAG (Directed Acyclic Graph) $G$ on $\{1,...,d\}$, which is the distribution $q$ factorizing with $G$ ...
0
votes
1answer
12 views

Compare convergence of optimization methods

I need to quantify how 2 optimization methods differ in convergence. When training a neural network I get the following plots, which show an error function after each gradient update. I think the ...
2
votes
1answer
82 views

How do CNN's avoid the vanishing gradient problem

I have been reading alot about convoloutional neural networks and was wondering how they avoid the vanishing gradient problem. I know deep belief networks stack single level auto-encoders or other ...
1
vote
0answers
20 views

Time series comparisons: early detection of mismatching series after n points for efficiency

I am doing time series comparisons. I have a set of values (my query set Q) that I need to compare against many other reference sets (R), each of which contains the same number of values as my query ...
0
votes
1answer
37 views

Update Rules in Expectation Maximization

I am emulating a certain PDF behaviour using a function. However, due to divergent improper integral, I don't have a closed form expression for the normalization constant. To get the PDF, I just ...
0
votes
0answers
37 views

A constrained Dynamic programming problem

I'm working on a linear time varying discrete(LTV) multi input multi output(mimo) system. I formulate the problem description in the following way $$x_i(k+1) = x_i(k)\cdot A_i(k) + B_i(k)\cdot ...
1
vote
0answers
37 views

Equivalence of the partial least square regresssion's iterative algorithm and its optimization problem

I am reading The Elements of Statistical Learning. This is a page from the partial least square section: The exercise asks to prove the equivalence between Algorithm 3.3 and Eq. (3.64). Here's my ...
4
votes
3answers
102 views

Decompose a time series into superposition of step functions?

Background I have time series data comprising hourly observations of a sensor's readings over a period of almost a year. The sensor records an environment whose baseline measurements should have ...
1
vote
0answers
12 views

Solving for primary variables of a linear program after already having solved for the dual

I was wondering if there is a general procedure of solving for the primary variables of a linear or quadratic or, in general, a convex program after already having solved the dual program. The ...
1
vote
0answers
13 views

Effect of dimensionality on gradient descent

How does increase in number of dimensions effect the convergence of gradient descent method. How is number of saddle points, local extrema related to dimensionality of data?
1
vote
0answers
29 views

The optimization problem of soft margin Support Vector Machine: How to interpret?

I try to understand what exactly we are trying to optimize in the case of Support Vector Machine problem, which supports soft margins. The original problem is posed first as, without soft margins ...
0
votes
0answers
34 views

Polynomial fitting for Shading Correction

I am reading the book Optimization for Computer Vision, and the first example of optimization is a regression for shading correction, in which the author proposes the following polynomial: $p_s(c,x) ...
1
vote
1answer
24 views

Is this problem convex ? (regularization term on xTw)

Suppose we want to solve the following: $$ \min_{w} f(x^Tw, y) + \lambda g(x^Tw) $$ with $f$ a (logistic) loss and $g$ something like a variance. Is this a convex optimization problem ? What are ...
0
votes
0answers
6 views

easy and difficult sample point

In my dataset, I can assume that there are two groups of samples. For one group, my model can easily learn the labels and for the other group my model has a poor performance in prediction of the ...
0
votes
0answers
8 views

Classification problem with constraints

I am trying to solve a classification problem with constraints and need advice on how I should approach it. Here's the problem: Given N observations, FLAG_j, j=1,..,N (this is a binar variable), and ...
0
votes
1answer
36 views

Dimension Reduction

I have a $N \times M$ matrix, the rank of matrix, $r$, is near to $min(M,N)$. I want to minimize the rank by removing some of the rows or columns to get $r << min(M,N).$ The goal is to achieve ...
0
votes
0answers
18 views

Bayesian optimization for known objective function with high dimension

I was wondering if one can use Bayesian Optimization algorithm for KNOWN, high-dimensional, expensive objective functions? If the answer is yes how efficient is that in terms of the quality of the ...
3
votes
0answers
167 views

Maximum likelihood estimation involving both probabilities and probability densities

Note: based on suggestions in the comments, I have rewritten this question. Please refer to the history for the original version. In general my question regards how to compute likelihoods in mixed ...
0
votes
0answers
46 views

Encouraging 'de-duplication' (ie: nullifying duplication) in an evolutionary optimization problem

I have a chronological production process with 24 cycles. Each production cycle has 5 of 8 conveyor belts in operation in each production cycle, whilst 3 of 8 belts are always at rest for ...
2
votes
0answers
8 views

Norm-bounded input

What is norm-bounded input? The expression is used in section 4.4 of 'Building High-level Features Using Large Scale Unsupervised Learning' by Le et al. I can find papers using the term, but not any ...
2
votes
1answer
71 views

How do I force the L-BFGS-B to not stop early? Projected gradient is zero

I'm trying to use the SciPy implementation of the fmin_l_bfgs_b algorithm using the following code: ...
2
votes
0answers
44 views

Unable to understand joint pdf and EM

I am unable to understand how the density function is derived in this paper Semiblind System Identification with Symbolic Chaotic Sequences The Authors have ...
1
vote
1answer
28 views

Unnatural clustering with known clusters shapes and optimization criteria

My question is similar to this question Clustering with shape prior, but with additional information. The second answer suggests a mixture model approach to this problem, which is something like ...
2
votes
1answer
61 views
0
votes
1answer
22 views

Time series tracking queue optimization problem

In order to track prices of many different products from different sources, I must optimally schedule a group of trackers dedicated to price collection (ie. collect one price at a time for each ...
0
votes
0answers
7 views

Optimisation of Fans In An Account

I have daily facebook data (past 3-4 months) of a company. I know how many fans they gained per day, fans they lost per day, engagements and so on. These are split into 'paid for', 'free from the ...
0
votes
0answers
32 views

optim() for multi variable returns values on the boundary in R

I would like to use function optim() in R to minimise the target function. The two optimised parameters both have constrains. I have created a test sampel data. ...
0
votes
1answer
36 views

SVM parameters clarification

James et al. in An introduction to the statistical learning (p. 351) claim that the solution to the support vector classifier problem involves only the inner products of the observations. They ...
2
votes
0answers
24 views

Perturbation of binned random variables

Setup: Assume that we have an interval coded random variable $T$ which takes values in the set of half-open intervals $\{[L_1, U_1), \ldots, [L_K, U_K]\}$, where $U_k$ and $L_k$ are the upper and ...
3
votes
1answer
45 views

Using quadratic programming to fit a piecewise linear model plus seasonality

I am reading this paper on fitting an L1TF model to data using quadratic programming. Section 7.4 states how one could add seasonality to the model however it doesn't go very far into it. I am trying ...
0
votes
1answer
26 views

Lagrange: Difference between minimizing and maximizing

The following seems straight forward: 1) Maximize f(x1, x2) = x1x2 subject to h(x1, x2) ≡ x1 + 4x2 = 16. Form the Lagrangian: L(x1, x2) = x1x2 − λ (x1 + 4x2 − 16) The first order conditions are (1) ...
0
votes
0answers
12 views

Faster gradient descent convergence by transforming the gradient?

If we modify the gradient descent update for a convex objective function $f(\boldsymbol{\theta})$ from $\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \nabla f(\boldsymbol{\theta}_t)$ to ...