Skip to main content

# Questions tagged [lagrange-multipliers]

The method of Lagrange multipliers finds critical points (including maxima and minima) of a differentiable function subject to differentiable constraints.

73 questions
Filter by
Sorted by
Tagged with
1 vote
0 answers
19 views

### Derivation of dual formulation of support vector regression

I'm trying to derive the dual formulation of epsilon-insensitive support vector regression. I think my derivation is correct, but I can't match it up to a result for the dual that I've seen given in ...
• 235
4 votes
1 answer
58 views

### Likelihood-ratio and score tests of a (non)linear combination of coefficients

The likelihood-ratio and score test are typically used for simple scalar hypotheses such as $\beta_1 = 0$ or $\beta_1 = \beta_2 = 0$. How can we test a linear combination of coefficients using the ...
• 1,824
1 vote
0 answers
36 views

### Support Vector Classifiers for Overlapping Classes

I am currently studying support vector classifiers (SVC), more specifically, the solution to the Lagrangian (Wolfe) dual function with the help of the book "The Elements of Statistical Learning&...
• 11
2 votes
2 answers
80 views

### How is the Representer theorem used in the derivation of the SVM dual form?

This is the primal form of the SVM hypothesis : $$h _{\mathbf{\vec w}, b}(\mathbf{\vec x}^{(i)}) = \mathbf{\vec w}\cdot \mathbf{\vec x}^{(i)} + b$$ The Representer theorem as formulated here ...
0 votes
0 answers
11 views

### Necessary condition for constrained optimization

Suppose $X=(X_1,\cdots,X_k)$ follows the multinomial distribution with a known size $n$ and an unknown probability vector $(p_1,\cdots,p_k)$. Find the necessary conditions for the solution to the ...
• 287
0 votes
0 answers
90 views

### Estimating the parameter of a Bernoulli distribution using probabilistic modeling and the MAP estimation

Suppose you tossed a coin multiple times. Sometimes you got heads and other times you got tails. You recorded your experiment in a dataset $X$. Now you want to estimate the parameter θ (which ...
1 vote
0 answers
41 views

### Why is there only a box constraint on alpha and not on mu when solving the dual problem of soft linear SVM?

I am currently learning about the linear SVM in the non-separable case. In the dual representation, we introduce the Lagrange multipliers μk and αk (see also this source: https://...
1 vote
0 answers
21 views

### Are solutions to the Lagrangian multipliers ($\alpha_i$) in a hard-margin SVM unique?

An intermediate step in the derivation of the hard-margin SVM's dual form is as follows: I also know that $a_i$ for all points not on the margin boundary is 0, which makes sense; they must be zeroed ...
1 vote
0 answers
28 views

### Min max formulation conversion to max min formulation. Reason?

Question is based on the screenshot attached. Based on paper here. I am not being able to understand why min max formulation (eq 4) is first converted to max min formulation (eq 5). Is it something ...
• 11
0 votes
0 answers
24 views

### Subspace test for multivariate normal distribution [duplicate]

Subspace test for multivariate normal distribution Suppose $X_1, X_2,\ldots, X_n$ are i.i.d. observations from a multivariate normal distribution $N(\mu,\Sigma)$ where $\Sigma$ is known. Furthermore, ...
• 1
1 vote
0 answers
77 views

### Can we convert the optimization of a loss function with regularization to the Lagrangian, constrained optimization *before* solving the optimization?

It is shown here that the optimization of a loss function with regularization, $$\text{argmin}_b L(X,b) + c ||b||_p \phantom{aaaaaaaaaaaaaaaaaaaaaaaa} (*)$$ is equivalent to the constrained ...
0 votes
0 answers
31 views

### Cannot understand the bound of Lagrangian parameter in SMO

I'm trying to understand SMO, but stuck to the part of bound for Lagrangian parameters. In the SMO paper(https://www.microsoft.com/en-us/research/uploads/prod/1998/04/sequential-minimal-optimization....
2 votes
1 answer
245 views

### How to maximize the ELBO in coordinate ascent variational inference

In the lecture by D.Blei: https://www.cs.princeton.edu/courses/archive/fall11/cos597C/lectures/variational-inference-i.pdf Variational inference is explained and he shows how to derive the optimal ...
• 449
0 votes
1 answer
49 views

### deriving the optimal distribution

Let the input variable $X \in \mathcal{X}$ and the target variable $Y \in \mathcal{Y}$. For a fixed hypothesis $h \in \mathcal{H}$ I want to solve \min_{p(X,Y)} \int_{\mathcal{X}}\...
• 127
1 vote
1 answer
38 views

### incorporating a distribution constraint in a minimisation objective

For a given (convex) hypothesis $h \in \mathcal{H}$, and the variables $X \in \mathcal{X}$ and $Y \in \mathcal{Y}$ I have the following optimisation problem: \min_{p(X,Y)} \int_{\...
• 127
1 vote
0 answers
138 views

### Derivation of the SVM regression problem

EDIT: The terms at the end will not get cancelled as they involve different variables: $\alpha$ and $\alpha^*$, etc. Now the post is corrected. I will leave this question here in case someone is as ...
• 269
0 votes
0 answers
235 views

### Why $\gamma$ in regularization term of XGBoost is defined as minimum loss reduction (not minimum squared loss reduction) and not substracted?

From the source https://xgboost.readthedocs.io/en/stable/tutorials/model.html I guess that the mean-squared error is optimized subjected to a constraint of minimum loss reduction. It appears like ...
• 11
3 votes
1 answer
97 views

### KKT Conditions for thresholds?

My main question is that when I use Lagrange Multipliers/KKT conditions to perform optimization with threshold constraints, I seem to get contradictory FOC. Here is a characteristic example: take an ...
• 115
1 vote
0 answers
234 views

• 1,116
3 votes
1 answer
3k views

### How to derive the $\ nR^2$ form of the LM test statistic?

Several lagrange multiplier (LM) test statistics =$\ nR^2$~$\chi^2$, where the LM test statistic is generated from regressing the square of residuals on some function in an auxiliary equation. The ...
• 83
0 votes
0 answers
72 views

• 41
1 vote
0 answers
30 views

### Score test using restricted mle

I am doing a study on Rao score test and still confused about the score vector. How am I supposed to calculate the score vector? Can I just use (ˆθn - θn) for the score vector? As I understood that ...
• 11
1 vote
1 answer
37 views

### Rewrite $\frac{1}{2}||x-u||_2^2$ subject to $||x||_1\le c$ to lagrangian form with multiplier $\lambda \ge 0$

Rewrite $\frac{1}{2}||x-u||_2^2$ subject to $||x||_1\le c$ to lagrangian form with multiplier $\lambda \ge 0$ So I'm pretty new to converting constraint functions to Lagrangian form, but I read that ...
• 720
0 votes
1 answer
101 views

### Non-negative Constraints in Soft-Margin SVM Lagrange Equation

I was reading the A Tutorial on Support Vector Machines for Pattern Recognition as a supplemental for my Intro to ML class and I wasn't sure why $a_i \geq 0$ and $\lambda_i \geq 0$ cannot be ...
• 1
0 votes
1 answer
291 views

2 votes
1 answer
340 views

### Maximum Entropy Discrete Distribution

In Pattern Recognition and Machine Learning the author uses Lagrange multipliers to find the discrete distribution with maximum entropy. Entropy is defined by; $$H=-\sum_i p(x_i)\ln(p(x_i))$$ and the ...
1 vote
0 answers
86 views

### What is $\alpha$ in SVM Lagrangian fucntions

Is it a column matrix or a row matrix? I saw on a CMU slide that $\alpha$ is defined as $\alpha=(\alpha_1, \alpha_2, ..., \alpha_n)^T>=0$? I dont see much of a formal definition for the exact form ...
1 vote
1 answer
46 views

### Convert the following expression w.r.t to the whole dataset instead of element of the dataset?

I am in the process of expressing the w in LSSVM with data points and constants. After I resolve the KKT conditions for the LSSVM I got $$w = \sum ^N _{i=1} \alpha_i x_i$$ Is it possible to convert ...
3 votes
1 answer
183 views

### Minimize $f(A,B)$ s.t. $\text{exp}(A)^T \text{exp}(B)=J_K$

I have a function $f(A,B)$ that maps a pair of two (tall) matrices, $A$ and $B$, to a scalar cost that I want to minimize. $A$ and $B$ both have $K$ columns. I also want to impose a set of equality ...
• 7,742
1 vote
0 answers
59 views

• 477
2 votes
2 answers
6k views

### Calculating the value of $b^{*}$ in an SVM

In Andrew Ng's notes on SVMs, he claims that once we solve the dual problem and get $\alpha^*$ we can calculate $w^*$ and consequently calculate $b^*$ from the primal to get equation (11) (see notes) ...
• 73
0 votes
0 answers
31 views

### If the Lagrangian formulation also has constraints, what is then the simplification?

Consider the following constrained optimization problem. $$\begin{array}{ll} \text{minimize} & f(w)\\ \text{subject to} & g_{i}(w) \leq 0\\ & h_{j}(w) = 0\end{array}$$ The equivalent ...
• 141
1 vote
0 answers
89 views

• 1,047